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Abstract 

This  study  reviewed  the  general  problem  o£  inadequate 
training  and  a  lack  of  selection  criteria  for  technical 
order  (TO)  managers  within  Aeronautical  Systems  Division 
(ASD).  A  questionnaire  was  administered  to  the  ASD 
technical  order  managers  in  an  attempt  to  determine  what 
they  thought  were  the  primary  areas,  within  the  training  and 
selection  criteria  arena,  that  needed  improvement.  Their 
opinions  on  how  to  best  improve  the  present  system  were  also 
solicited.  Telephone  interviews  were  also  conducted  with 
six  defense  contractors  in  an  attempt  to  broaden  the  input 
on  how  best  to  improve  the  present  ASD  technical  order 
manager  training  and  selection  process.  Based  on  the 
questionnaire  and  telephone  interview  responses, 
recommendations  were  made  for  improving  the  present  process. 
These  recommendations  Included  the  establishment  of:  a 
minimum  grade  criteria,  a  minimum  writing  ability  criteria 
(with  a  training  option),  prior  experience  guidelines  (with 
a  training  option  for  hands-on  experience),  a  career  channel 
for  TO  managers  to  follow,  improved  continuity  objectives, 
mandatory  training  classes,  improved  personal  relations 
emphasis,  regular  review  of  training  quotas,  and  an 
apprenticeship  program.  t 
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AN  ANALYSIS  OF  THE  SELECTION  CRITERIA  AND  TRAINING  FOR 


AERONAUTICAL  SYSTEMS  DIVISION  TECHNICAL  ORDER  MANAGERS 

I.  Introduction 

This  chapter  introduces  the  general  problem  of 
inadequate  training  and  a  lack  of  selection  criteria  for 
technical  order  managers  within  Aeronautical  Systems 
Division.  The  research  objectives  are  defined  with  the 
investigative  questions.  These  questions  are  answered  via 
the  measurement  questions  which  are  addressed  in  this 
chapter  as  well  as  the  scope  and  introductory  background  of 
the  problem. 

General  Issue 

Technical  orders,  or  TO's  as  they  are  commonly  called, 
are  critical  to  the  maintainability  and  supportability  of 
every  Air  Force  weapon  system.  The  person  selected  to 
manage  these  important  publications  should  be  qualified, 
have  the  best  background,  and  best  training  possible. 

Specific  Problem 

The  people  in  charge  of  managing  the  technical  orders, 
the  technical  order  managers,  or  TOMA's  as  they  are  commonly 
called,  do  not  always  have  the  best  experience  or  training 
to  qualify  them  to  manage  the  complex  job  requirements  of 
being  a  technical  order  manager.  In  fact,  the  Air  Force  has 
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no  specific,  formal  qualification  requirements  nor  specific 
selection  and  training  criteria  for  technical  order 
managers.  There  is  no  distinct  Air  Force  Specialty  Code 
(AFSC),  nor  is  there  a  distinct  Federal  Service  Career 
Series  for  this  job. 

Scope 

The  problems  with  technical  order  manager  selection  and 

training  appear  to  be  Air  Force  wide.  One  of  the  problems 

is  that  the  technical  order  managers  are  often  assigned  too 

late  in  the  program.  It  is  very  Important  from  the 

continuity  standpoint  that  the  technical  order  managers  be 

identified  early  in  the  program. 

. . .  the  opportunity  to  influence  the  technical  order 
acquisition  process  is  diminished  as  engineering  design 
and  the  acquisition  strategy  is  solidified.  The  key  to 
developing  quality  publications  at  competitive  cost  is 
to  allow  the  process  to  demonstrate  its  effectiveness 
in  improving  overall  system  supportabi 1 ity .  Very 
often,  the  technical  order  manager  is  not  yet  assigned 
during  this  process.  (6:40) 

Another  problem  with  the  technical  order  managers  is 
the  informal  selection  criteria,  i.e.,  they  are  often  hired 
straight  off  the  flight  line  based  on  their  hands-on 
experience  using  the  technical  orders.  This  hiring 
practice  coupled  with  the  lack  of  specific  training 
requirements  for  the  new  technical  order  managers  plays 
havoc  with  an  already  complicated  acquisition  system  -  a 
system  which  includes  negotiating,  working  with  contracting, 
preparing  for  program  management  responsibility  transfer 
(PMRT)  and  the  associated  follow-on  depot  care  relationship 
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with  Logistics  Command,  as  well  as  required  familiarity  with 
numerous  forms  unique  to  the  acquisition  field. 

This  thesis  will  recommend  specific  actions  to  be  taken 
in  order  to  improve  the  present  situation.  The  problem  has 
been  recognized  and  pointed  out  for  many  years;  however,  no 
significant  Air  Force  wide  change  has  occurred.  A  change 
would  have  more  likelihood  of  occurring  in  a  smaller 
organization  than  a  larger  one  (i.e..  Aeronautical  Systems 
Division  (ASD)  rather  than  Air  Force).  Therefore,  this 
study  will  focus  on  and  make  specific  recommendations  for 
ASD  only.  If  some  of  the  recommendations  proposed  herein 
were  to  be  implemented  and  proved  to  be  successful  in 
enhancing  the  scope  of  training,  selection  criteria,  and 
ultimately  the  performance  of  the  ASD  technical  order 
managers,  it  is  hoped  that  further  implementation  options 
would  be  investigated. 

For  a  comparison  and  as  a  compliment  to  the  information 
obtained  from  the  ASD  technical  order  managers  themselves, 
six  large  aerospace  contractors  which  produce/develop 
technical  orders  in  high  volume  were  contacted  by  telephone 
for  interviews  on  how  they  select  and  train  their  technical 
order  managers.  This  was  done  to  provide  another  viewpoint 
in  the  development  of  an  approach  to  training  and  selection 
criteria  for  technical  order  managers.  It  was  felt  that  an 
outside  perspective  might  help  avoid  both  the  pitfalls  of 
tunnel  vision  and  the  natural  tendency  to  maintain  the 
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status  quo  -  proclivities  which  could  easily  beset  strictly 
Internal  studies. 

One  aspect  of  perhaps  improving  the  selection  criteria 
issue  is  the  code  identifier  added  on  to  the  Air  Force 
Specialty  Codes  and  job  series'  of  those  persons  who  have 
worked  with  technical  orders.  This  code  identifier  has  been 
recommended  to  help  the  personnel  systems  more  easily 
identify  all  persons  with  technical  order  user  experience  so 
that  a  pool  of  such  people  could  be  maintained  in  order  to 
select  the  best  qualified  people  for  the  technical  order 
manager  jobs.  It  was  felt  that  the  background  and  research 
required  in  the  area  of  personnel  (both  llltary  and 
civilian)  to  address  this  issue  was  too  vast  to  attempt  to 
include  this  aspect  in  this  thesis.  This  approach  to 
improving  the  selection  criteria  is  a  recommendat ion  for 
further  research. 

Investigative  Questions 

In  order  to  address  the  general  issue  of  selection 
criteria  and  training  for  ASD  technical  order  managers  the 
following  two  investigative  questions  must  be  answered: 

1.  What  selection  criteria  should  the  Aeronautical 
Systems  Division  establish  for  technical  order 
managers? 

2.  What  training  should  the  Aeronautical  Systems 
Division  provide  for  technical  order  managers? 
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Measurement  Questions 


The  following  questions  were  asked  via  questionnaire 
and  telephone  interviews  in  order  to  answer  the 
Investigative  Questions: 

ASP  Technical  Order  Manager  Questions. 

1)  What  is  your  grade? 

2)  What  is  the  highest  level  of  education  you  have  achieved? 

3)  How  long  have  you  been  working  as  a  technical  order 
manager? 

4)  What  is  the  total  dollar  value  of  the  tech  orders  you 
manage? 

5)  Approximately  how  many  technical  orders  do  you  manage? 

6)  In  what  capacity  have  you  worked  with  tech  orders  prior 
to  your  present  job  (flight  line,  maintenance,  in  the  System 
Program  Office,  etc.)? 

7)  What  kind  of  technical  order  acquisition  manager-related 
training  have  you  had? 

8)  What  type  of  experience  did  you  have  to  qualify  you  for 
this  job? 

9)  What  kind  of  experience,  prior  to  your  coming  on  the  job, 
would  have  better  prepared  you  for  the  job  you  have  now? 

10)  What  kind  of  training  would  you  still  like  to  have  in 
order  to  improve  your  present  level  of  job  performance? 

11)  What  shortcomings  have  you  witnessed  in  other  technical 
order  managers? 

12)  What  strong  points  have  you  witnessed  in  other  technical 
order  managers? 

13)  Please  try  to  list  and  explain  If  necessary  all  of  your 
duties  associated  with  tech  order  management  (such  as 
negotiations,  in-process-reviews,  data  item  description 
reviews,  etc .  )  . 

14)  Why  do  you  think  you  were  selected  for  this  job? 
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15)  Is  there  anything  that  you  would  like  to  add  that  might 
help  improve  the  training  of  technical  order  managers?  If 
so,  please  elaborate. 

16)  What  do  you  think  should  be  included  in  the  criteria 
for  selection  of  technical  order  managers?  (Examples:  field 
experience.  Systems  230,  negotiation  experience,  etc.) 

Contractor  Questions. 

1)  How  do  you  select  your  TO  managers  (or  publications 
managers ) ? 

2)  Are  there  selection  criteria?  If  so,  what  are  they? 

3)  What  type  of  background  for  a  TO  (or  publications) 
manager  have  you  found  to  be  most  successful? 

4)  What  type  of  training  do  you  provide  for  your  TO 
managers? 

5)  What  experience  or  training  do  you  feel  is  most 
essential  to  the  success  of  a  To  manager? 

6)  What  are  the  weak  points  of  your  current  selection  and 
training  methods?  Strong  points? 


Background 

In  order  to  competently  address  the  issue  of  technical 

order  managers,  a  basic  understanding  of  the  weapon  system 

acquisition  life  cycle  as  well  as  the  technical  order 

acquisition  cycle  and  how  it  fits  in  with  the  weapon  system 

acquisition  life  cycle  is  required.  The  weapon  system 

acquisition  life  cycle  or  process  normally  includes  concept 

exploration,  demonstration  and  validation,  full  scale 

development,  and  production  and  deployment. 

The  technical  order  development  cycle  is  divided  into 
four  broad  and  general  phases.  These  are  planning  and 
task  identification,  preparation  and  development, 
printing  and  distribution,  update  and  changes.  The 
planning  and  development  of  technical  orders  prescribes 
a  development  cycle  of  20  to  24  months  depending  on 
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factors  such  as  complexity  of  equipment/  maintenance 
concept,  text  format  requirements,  etc.  (6:39) 

"Planning  for  the  acquisition  of  technical  orders 

should  begin  with  the  using  command  and  the  preparation  of 

the  'Statement  of  Need'"  (6:39).  The  Statement  of  Need 

(SON)  is  the  document  that  gets  the  whole  weapon  system 

acquisition  life  cycle  started.  If  the  need  is  verified, 

then  the  first  phase,  concept  exploration,  is  initiated. 

Sometime  during  the  production  and  deployment  phase, 

program  management  responsibility  transfer  (PMRT)  occurs. 

This  transfer  of  responsibility  from  Systems  Command  to 

Logistics  Command  happens  earlier  for  less  complex 

aeronautical  subsystems  and  armament  programs,  midway  for 

less-than-ma jor  programs  such  as  simulators  and  tactical 

missiles  and  later  in  the  production  and  deployment  phase 

for  major  programs. 

Prior  to  PMRT,  Systems  Command  is  in  the  "driver's 
seat",  directing  the  acquisition  management  tasks.  At  a 
point  where  these  tasks  are  essentially  complete  and  a 
timely,  orderly  and  clean  transfer  of  responsibility  (4:28- 
1)  can  take  place,  PMRT  occurs.  This  involves  technical 
orders  also.  Most  of  the  acquisition  tasks  should  be 
completed,  and  with  the  exception  of  a  few  residual  tasks. 
Logistics  Command  takes  over  responsibility  for  the 
technical  orders  also.  Logistics  Command  is  then  the 
"driver"  for  the  depot  care  of  maintaining  and  supporting 
the  weapon  system.  However,  should  a  major  modification  be 
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directed.  Systems  Command  would,  in  most  cases,  be  assigned 
the  responsibility  for  buying  and  managing  the  development 
of  the  major  modification. 

Such  a  major  modification  could  be  a  Class  V  (read 
"Class  Five")  modification  which  is  the  installation  or 
removal  of  equipment  changing  the  mission  capability  of  the 
weapon  system. 

Assignment  of  program  responsibilities  in  Class  V 
modification  acquisition  depends  on  the  nature  of  the 
modification  and  on  program  management  responsibility 
( PMR )  for  the  system  and  equipment  to  be  modified. 

Most  often,  AFSC  will  be  responsible  for  developing  the 
equipment  providing  the  new  capability.  The 
implementation  of  the  Class  V  modification  will  then  be 
assigned  to  the  command  with  PMR  for  the  system  being 
modified.  Because  most  Class  V  modifications  are 
directed  after  program  management  responsibility 
transfer  ( PMRT ) ,  AFLC  is  most  often  the  implementing 
command.  The  most  likely  assignment  of 
responsibilities  in  program  patterns  ...  would  be  for 
AFSC  to  manage  the  development  program  and  AFLC  to 
manage  the  implementation  program.  (4:35-1,35-2) 

The  quality  of  communication  and  management  skills 

exhibited  in  the  relationship  between  the  Systems  Command 

technical  order  manager  and  the  Logistics  Command  technical 

order  manager  prior  to,  during,  and  after  PMRT  is  critical 

to  the  success  of  the  updates  to  existing  manuals  occurring 

on  time  and  at  a  reasonable  cost.  Knowledge  of  the 

procedures,  requirements  and  status  of  PMRT  is  essential. 

Several  studies  have  cited  the  inadequacies  of  the 

technical  order  management  world  (1,2,8).  Minor  progress 

has  been  made  in  addressing  these  inadequacies;  however, 

there  is  still  ample  room  for  improvements,  especially  in 


8 


the  areas  of  technical  order  manager  selection  and  training 
criteria. 

Fortunately,  higher  management  is  aware  that  there  are 
personnel  shortcomings.  The  ground  is  ripe  for  making  solid 
recommendations  for  improvements  to  the  process  of 
developing  a  quality  technical  order  manager  workforce. 
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II.  Literature  Review 


Technical  orders  are  Air  Force  instructions  issued 
through  the  Air  Force  Technical  Order  ( AFTO )  system  for  the 
operation  and  maintenance  of  all  Air  Force  equipment  (6:19). 
The  AFTO  system  provides  data  required  by  the  Air  Force  for 
the  operation  and  maintenance  of  all  equipment  and  ensures 
availability  of  this  information  where  and  when  needed 
(6:15).  The  importance  of  this  technical  data  cannot  be 
underscored  enough.  Williams  and  Winn,  in  their  1980 
master's  thesis  tied  it  together  as  follows: 

While  it  may  be  obvious  that  proper  weapon 
maintenance  is  necessary,  what  may  not  be  so  obvious  is 
the  role  that  is  played  by  the  technical  data  and,  more 
specif ically,  the  technical  order.  The  technical  order 
is  where  the  maintenance  procedures  are  standardized 
into  a  body  of  knowledge.  When  maintaining  complex 
systems,  especially  with  an  increasingly  younger  and 
inexperienced  work  force,  standard  methods,  procedures, 
and  instructions  are  vital.  Conceptually,  one  can 
visualize  a  hierarchy  of  requirements,  starting  from 
the  requirement  of  National  Security,  to  a  strong  Armed 
Force,  to  an  effective  weapon,  to  adequate  maintenance 
of  that  weapon  system  and,  finally,  to  the  requirement 
for  accurate  and  adequate  maintenance  procedures 
supporting  that  maintenance.  (10:2-3) 

It  follows  then,  with  the  very  high  criticality  affixed  to 

the  adequacy  and  accuracy  of  technical  orders,  that  the 

person  charged  as  responsible  for  acquiring  them  would  be 

likewise  viewed  and  treated  as  a  critical,  valued  resource. 

The  organization  or  individual  assigned  technical  order 

acquisition  responsibility  is  called  the  TOMA  (4:38-1). 

Although  TOMA  actually  stands  for  Technical  Order  Management 

Agency,  the  acronym  is  used  Interchangeably  to  refer  to  the 
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responsibilities  of  an  organization  or  an  individual  as 
described  above. 

The  TOMA  is  a  staff  functional  manager  appointed  by  the 
weapon  system  program  manager  (PM)  or  Deputy  Program  Manager 
for  Logistics  ( DPML )  early  (hopefully)  in  the  program.  The 
TOMA  will  advise  the  PM  or  DPML  in  a  key  functional  area  - 
that  of  technical  orders. 

The  TOMA ' s  responsibilities  are  extensive.  Simply 

put,  the  TOMA  "must  procure  the  most  accurate,  adequate  and 

cost  effective  manuals  possible  within  the  resources 

available"  (5:2-6).  (Manuals,  or  technical  manuals  [TMs], 

are  a  type  of  TO.)  These  responsibilities  include: 

-initiating  and  coordinating  pre-contract  planning  for 
procurement 

-preparing,  coordinating,  and  distributing  a  Technical 
Manual  Management  Plan 

-preparing  inputs  to  the  Statement  of  Work 

-conducting  and  chairing  all  technical  manual  (TM) 
conferences,  meetings,  reviews,  and  other  joint  agency 
efforts 

-coordinating  with  the  using  and  supporting  commands  to 
ensure  participation  in  all  TM  acquisition  activities 

-overseeing  a  TM  quality  assurance  program 

-participating  in  factfinding/negotiations 

-ensuring  that  contractors  develop  TMs  in  the  most  cost 
effective  manner 

-reviewing  Support  Equipment  Recommendation  Data  (SERD)  for 
any  TM  Impacts 

-providing  inputs  to  program  budgets  for  TM  development 
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-authorizing,  with  the  mutual  consent  of  the  using  and 
supporting  commands,  the  use  of  verified  preliminary  TO's 
(PTOs)  (5:2-7, 2-8). 

With  responsibilities  like  these,  the  job  of  a  TOMA 
obviously  requires  varied  and  extensive  qualifications.  The 
Air  Force  Institute  of  Technology  (AFIT)  Systems  230  course 
text  describes  the  characteristics  of  a  proficient  TOMA  as 
follows : 


Ideally,  the  TOMA  should  be  a  highly  qualified  analyst, 
have  some  knowledge  of  configuration  management,  be  a 
proficient  technician  and  an  experienced  planner,  have  an 
understanding  of  contracting  and  appropriate  Defense 
Acquisition  Regulation/Federal  Acquisition  Regulation 
(DAR/FAR)  clauses  as  they  apply  to  TOs,  be  a  proficient 
negotiator  and  coordinator,  and  most  of  all,  the  TOMA  should 
be  an  efficient  manager  (6:38). 

The  criticality  of  the  TOMA  is  even  more  apparent  in 
other  TOMA  responsibilities  cited  in  the  literature. 

According  to  the  joint  Air  Force  Logistics  Command 
(AFLC)/Air  Force  Systems  Command  (AFSC)  Pamphlet  800-34, 

"The  major  obstacle  to  successful  TO  management  is  failure 
to  assign  the  TOMA  early  in  the  program.  This  results  in 
inadequate  planning"  (4:38-4).  The  AFLC/AFSC  Pamphlet  goes 
on  to  say  how  this  inadequate  planning  can  lead  to  expensive 
problems,  especially  since  TO  cost  estimates  for  major 
programs  are  in  the  $600  -  $900  million  range. 

Consequently,  TO  acquisition  deserves  full  managerial 
attention  (4:38-4). 

Williams  and  Winn,  in  their  1980  Master's  Thesis,  give 
examples  of  specific  problems  associated  with  inadequate 
TO's  at  the  user  organization.  These  examples  include 
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unnecessary  maintenance  manhours  to  research  corrections 
needed,  technicians  unable  to  perform  the  task  as  directed 
by  the  manuals,  inaccurate  data  and  increased  maintenance 
manhours  expended  in  troubleshooting  equipment  deficiencies 
(10:11-12) . 

Given  the  criticality  of  TOMA  responsibilities  and  the 
high  dollar  value  of  the  TO's,  one  would  expect  a  very 
sophisticated  selection  process  for  TOMAs.  Unfortunately, 
that  is  not  the  case  at  all  as  several  researchers  have 
discovered.  Timothy  N.  Towner,  in  his  1983  master's  thesis, 
noted  this  shortfall  as  well  as  an  additional  problem  of 
personnel  turnover  within  the  TOMA  ranks:  "...  there  are 
no  specific  job  descriptions  or  qualifications  for  TOMA 
personnel,  and  the  TOMA  is  susceptible  to  rapid  turnover  of 
personnel"  (8:39).  Billingham  and  Klassen,  in  their  1985 
master's  thesis  found  a  similar  deficiency  regarding  how  TO 
managers  are  identified: 

There  presently  exists  no  single  standard  or 
universal  method  within  the  Air  Force  Systems  Command 
to  identify  TO  managers.  Each  product  division  has 
unique  requirements  for  performance,  yet  none  of  the 
product  divisions  has  developed  a  method  to  Identify 
those  individuals  who  could  meet  their  performance 
requirements.  Factors  such  as  having  previously  been  a 
user  level  maintenance  technician,  while  applicable  in 
all  product  divisions,  were  not  a  common  factor 
possessed  by  all  TO  managers  within  any  product 
division.  Experience  as  a  TO  manager,  a  key  point  of 
consideration  in  any  selection  process,  is  not  even  an 
identifiable  skill  under  the  present  military  personnel 
system.  (1:75-76) 

The  present  system  it  appears,  is  not  much  better:  Mr. 
Chandler  "Chuck"  Weaver  is  an  RJO  Enterprises,  Inc.  support 
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contractor  working  for  ASD/ALXS,  (the  researcher's 
sponsoring  organization  and  the  office  which  is  trying  to 
overhaul  the  present  state  of  training  and  selection 
criteria  for  ASD  technical  order  managers),  who  has  20  years 
of  active  duty  Air  Force  experience  plus  14  years  of  civil 
service  experience  with  over  20  years  in  technical  orders. 

He  says  from  what  he  has  observed  in  the  "real  world", 
generally  speaking,  when  a  technical  order  manager  is 
needed,  one  is  usually  obtained  by  word  of  mouth.  The 
requiring  program  manager  (PM)  or  Deputy  Program  Manager  for 
Logistics  (DPML)  will  ask  around  at  the  different  program 
offices  for  other  TOMA's  interested  in  changing  programs. 

If  there  are  no  takers,  the  PM  or  DPML  will  then  request 
(from  the  civilian  personnel  office)  a  listing  of  civilians 
in  the  346  (logistics  management  specialist)  or  1670 
(equipment  specialist)  job  series  from  which  to  select  a 
technical  order  manager.  Or  if  a  military  technical  order 
manager  is  desired,  the  PM  or  DPML  will  go  through  the 
military  personnel  system  requesting  a  technical  order 
manager.  Generally  the  next  person  through  the  military 
personnel  door  with  some  kind  of  maintenance  experience 
(usually  aircraft,  electronic,  or  avionic  equipment 
technicians,  and  some  support  equipment  technicians)  will  be 
assigned  to  the  requesting  SPO.  If  the  need  is  immediate, 
often  the  PM  or  DPML  will  choose  a  military  overage  assigned 
to  the  4950th  Test  Wing  in  Area  C  of  Wr ight-Patterson  Air 
Force  Base  ( 9 ) . 
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In  addition  to  the  insufficient  selection  methods  and 


criteria,  maintaining  continuity  of  the  technical  order 
manager  also  is  a  major  problem  caused  by  high  turnover  of 
personnel.  Several  sources  point  out  the  importance  of 
having  unbroken  continuity  of  the  TOMA  function.  AFLC/AFSC 
Pamphlet  800-34  states,  "The  TOMA  also  acts  as  source 
selection  item  evaluator  for  the  TO  portion  of  the  RFP 
(Request  For  Proposal).  Continuity  is  important  here.  The 
TOMA  who  wrote  the  TO  portion  of  the  RFP  is  best  qualified 
to  evaluate  the  proposal  responses"  (4:38-1,38-2).  The  AFIT 
Systems  230  (SYS  230,  Air  Force  Technical  Order  Acquisition 
and  Management)  course  text  also  cites  continuity  problems 
with  the  TOMA,  especially  military  TOMA's  due  to  their 
automatic  reassignment  rotation  back  to  their  "real"  job 
areas . 

Continuity  is  always  a  problem  for  TOMAs.  Military 
personnel  generally  are  assigned  to  specific  program 
TOMAs  for  tours  of  duty  of  specified  duration;  when  the 
tour  of  duty  is  over,  they  will  be  reassigned 
elsewhere.  Assignment  of  military  personnel  ordinarily 
has  little  relationship  to  a  program  acquisition 
schedule.  (6:37-38) 

The  TOMA's  responding  to  a  telephone  survey  in  Billlngham 
and  Klassen's  research  said  it  takes  12  to  18  months  to 
become  a  proficient  TOMA.  Almost  50  percent  of  the  TOMA's 
interviewed  had  less  than  two  years  experience  (1:79). 
Clearly,  continuity  is  a  problem. 

Fortunately,  the  problems  with  the  TOMA  identification 
process  and  the  continuity  of  the  TOMA  have  not  gone 
unnoticed.  Unfortunately,  those  that  have  noticed  have  not 
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been  in  a  power  position  to  do  something  about  it.  Those 

who  are  in  a  "do  something"  position  are  usually  only  aware 

of  personnel  problems  in  a  general  sense.  In  a  1979 

interview  with  then  Colonel  Richard  F.  Gillis,  CX  Deputy 

Program  Manager  for  Logistics,  ASD/AFHL  (now  Major  General 

Gillis,  Commander,  Warner-Robins  Air  Logistic  Center), 

Colonel  Gillis  stated  there  is  "a  requirement  for  more 

qualified  AFLC  logisticians  upon  the  establishment  of  the 

System  Program  Office  (SPO)  cadre"  (10:10-11).  Even  higher 

level  reports  point  out  the  problem  of  quality  of 

acquisition  personnel.  The  1986  Packard  Commission  Report 

says,  "Our  study  convinces  us  that  lasting  progress  In  the 

performance  of  the  acquisition  sys*-  demands  dramatic 

improvements  in  our  management  of  acquisition  personnel  at 

all  levels  within  (the)  DoD  (Department  of  Defense)"  (7:65- 

66).  The  Packard  Report  also  touched  on  the  importance  of 

training:  ".  .  .  it  is  vitally  important  to  enhance  the 

quality  of  the  defense  acquisition  workforce  -  both  by 

attracting  qualified  new  personnel  and  by  improving  the 

training  and  motivation  of  current  personnel"  (7:65-66).  In 

July  1989,  Secretary  of  Defense,  Dick  Cheney,  submitted  a 

report  to  the  President  recommending  improved  management  of 

military  and  civilian  personnel. 

A  dedicated  corps  of  acquisition  officers  will  be 
established  in  each  of  the  Services,  including  distinct 
sub-specialties  in  systems  development,  procurement  and 
logistics.  The  Secretaries  of  the  Military  Departments 
will  present  detailed  plans  for  these  purposes  in 
October  1989.  (3:4) 
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Secretary  Cheney  also  emphasized  the  importance  of 
continuity  in  terms  of  cost,  an  important  item  on  every 
agenda.  "Reliable  planning,  funding,  and  system 
configuration,  and  continuity  in  management  personnel, 
greatly  increase  the  likelihood  that  systems  will  be 
delivered  on  time  and  at  projected  cost"  (3:10). 

A  logical  solution  for  addressing  the  continuity  and 
ldentif ication  problems  of  TOMA’s  would  be  to  train  a  pool 
of  qualified  people  and  install  them  in  TOMA  jobs  for  four 
year  minimum  tours.  Unfortunately,  this  area  in  the  TOMA 
realm  has  its  inadequacies  also.  Brown  and  Lyon,  in  their 
1984  Master's  Thesis,  determined  through  a  telephone  survey 
that  82  percent  (of  130  polled)  believed  the  "training  and 
knowledge  level  of  TO  acquisition  personnel  was  not 
adequate”  (2:64).  Indeed,  though  the  AFIT  Systems  230 
course  text  has  a  section  entitled,"  Training  and  Experience 
Requirements"  (for  the  TO  manager),  there  are  no 
requirements  listed,  only  some  typical  backgrounds  of  TOMA's 
(6:37-38).  Even  the  Systems  230  course  itself.  Air  Force 
Technical  Order  Acquisition  and  Management,  is  not  a 
prerequisite  to  becoming  a  TOMA.  Billingham  and  Klassen's 
research  in  1985  showed  that  less  than  28  percent  of  the  101 
TO  managers  interviewed  had  taken  the  Systems  230  course 
(1:79-80)  . 

The  majority  of  the  respondents  stated  their  only 
previous  TO  experience  was  as  user  level  maintenance 
technicians.  These  findings,  combined  with  the  fact 
that  TO  managers  have  no  extensive  training  in 
management  principles,  support  the  conclusion  that 
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training  can  improve  the  effectiveness  of  the  TO 

manager.  (1:80) 

Billingham  and  Klassen  identified  five  key  elements  for 
Influencing  selection  criteria  in  their  master's  thesis: 
formal  management/training,  prior  TO  experience,  prior 
management  experience,  size  and  complexity  of  program,  and 
acquisition  phase  when  selection  of  a  TO  manager  is  needed 
(1:93).  They  also  made  the  following  further  recommenda¬ 
tions  for  research:  (1)  Study  APLC  TO  management  personnel 
(they  only  studied  AFSC  personnel).  (2)  Establish  some 
standard  of  measure  to  judge  and  compare  TO  cost,  timeliness 
and  useability  across  acquisition  programs.  (3)  Look  at 
what  factors  contribute  to  the  lack  of  action  toward 
improving  the  evidenced  problems  (1:95). 

Chapter  Summary 

The  preceding  paragraphs  have  established  the 
criticality  and  costliness  of  technical  manuals  and  the 
corresponding  criticality  of  the  individual  or  agency,  the 
TOMA,  responsible  for  acquiring  these  technical  manuals. 
Three  major  deficiencies,  insufficient  selection  criteria, 
lack  of  continuity,  and  inadequate  training  requirements 
have  been  discussed  regarding  the  technical  order  manager. 
The  point  was  also  made  that  higher  level  management  is 
aware  of  the  general  inadequacies  in  our  personnel  policies, 
if  not  specifically  regarding  the  TOMA,  at  least  regarding 
acquisition  personnel  in  general,  and  all  agree  that  steps 
need  to  be  taken  to  Improve  present  policies.  Given  higher 
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management's  present  attitude  and  awareness,  the  ground  is 
certainly  ripe  for  good  recommendations  and  subsequent 
improvements  in  a  presently  lacking  system.  Using  the 
foregoing  research  and  information,  and  by  surveying  ASD  and 
non-government  technical  order  managers,  plausible  selection 
criteria  as  well  as  specific  training  requirements  for  ASD 
technical  order  managers  can  be  recommended  for  great 
savings  to  the  government  in  terms  of  dollars  and 
irritation.  This  will  ultimately  lead  to  more  efficient  and 
effective  use  of  tax  payers'  dollars  as  well  as  a  stronger 
national  defense  posture. 
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Methodolo 


This  chapter  describes  the  procedures  used  in  the 
descriptive  study  to  determine  who  are  the  ASD  technical 
order  managers  (in  terms  of  grade,  education  and 
experience),  what  are  their  responsibilities  (in  terms  of 
dollar  value  and  number  of  technical  orders  managed  as  well 
as  requirements  involving  negotiations,  data  item 
descriptions,  logistics  support  analysis,  etc.)  and  how  the 
Air  Force  can  better  prepare  them  to  do  their  jobs.  The 
research  method  used  is  justified,  the  instrument  is 
explained,  the  populations  are  defined  and  the  data 
collection  and  statistical  tests  are  addressed. 

Justification  of  Research  Method 

The  survey  approach  was  used  in  this  research  in  order 
to  obtain  ideas  on  how  to  improve  the  training  and  selection 
criteria  for  tech  order  managers .  The  data  was  obtained 
from  the  ASD  technical  order  managers  via  mail  surveys. 

Mall  surveys  were  considered  optimum  because  rf  the  large 
population  (84  technical  order  managers  versus  one 
researcher),  the  added  anonymity  (respondents  would  tend  to 
be  more  honest),  and  the  greater  accessibility  (many  of 
these  technical  order  managers  are  travelling  almost 
constantly).  It  was  also  thought  that  the  mail  surveys 
would  give  the  respondents  more  time  to  think  about  the 
issues  addressed,  discuss  the  problems  with  their  co-workers 
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and  take  the  time  to  verify  their  facts,  such  as  the  dollar 
value  of  the  technical  orders  they  manage  -  something  they 
might  not  readily  know.  All  of  these  issues  clearly 
justified  a  mail  survey.  By  contrast,  the  contractor  data 
was  obtained  via  telephone  interviews.  The  telephone 
interviews  were  the  most  economical  and  reliable  method, 
given  that  the  respondents  were  from  all  over  the  country 
and  that  opinions  and  general  information  about  past 
experience  were  required.  The  difference  in  population 
sizes  between  the  Air  Force  respondents  and  the  contractors 
was  considered,  however,  it  is  not  a  factor  because 
contractor  responses  comprised  merely  a  collection  of  ideas 
and  were  not  intended  to  be  a  representative  sample  of  all 
the  defense  contractors'  approaches  and  practices.  It  was 
felt  that  telephone  interviewing  would  greatly  decrease  the 
response  time.  It  was  also  felt  that  the  personal  approach 
would  be  more  effective  than  a  mail  survey  given  that  the 
respondents  were  recommended  by  either  Mr.  Arthur  Munguia, 
thesis  advisor,  who  knew  them  personally,  or  were  suggested 
by  the  contractors  themselves  who  were  contacted  for 
interviews . 

Instrument 

There  were  16  measurement  questions  asked  of  the  ASD 
technical  order  managers  (see  page  5).  Questions  1  through 
8  were  essentially  background  questions  which  asked  for 
conscious  surface  (known  automatically,  e.g.,  rank)  and  some 
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subsurface  level  (might  take  some  minimal  thought  or 
calculation,  e.g.,  the  dollar  value  of  the  TO's  one  manages) 
data.  These  questions  were  asked  in  order  to  establish  the 
experience,  training,  and  technical  order  responsibility  (in 
terms  of  numbers  and  dollars)  of  the  TO  managers. 

Billingham  and  Klassen  (1)  established  the  lack  of  training 
and  selection  criterion  for  technical  order  managers  in 
their  1985  master's  thesis.  These  eight  questions  were 
asked  in  order  to  confirm  the  results  of  their  research  as 
well  as  the  hypothesis  of  this  thesis  (inadequate  training 
and  selection  criterion  for  technical  order  managers). 

Questions  1  through  5  were  structured  questions  with 
choices  listed  from  which  the  respondent  could  choose  just 
one  answer.  Question  6  was  an  open  question  with  examples 
of  suggested  answers  listed  to  give  the  respondent  an  idea 
of  what  was  been  sought.  Questions  7  through  12  were 
multiple  choice  questions  with  an  "Other"  category  given  if 
the  choices  were  not  sufficient.  The  "Other"  category  was 
to  be  elaborated  upon  at  the  end  of  the  questionnaire. 

Questions  9  through  16  were  opinion  and  judgment 
questions  calling  for  reflective  thought  and  analysis  of  all 
the  issues  and  aspects  of  the  job.  These  questions  are 
really  the  crux  of  the  questionnaire  and  research,  i.e., 
what  does  the  technical  order  manager  think  is  the  most 
essential  in  terms  of  training  and  past  experience  in  order 
to  best  do  their  job? 
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Question  13  (Try  to  list  and  explain,  if  necessary,  all 
of  your  duties  associated  with  technical  order  management.) 
was  an  open  question  with  response  examples.  This  question 
was  a  "catch-all"  attempt  to  ensure  that  all  areas  of 
responsibility  are  being  considered  in  planning  for  training 
and  selection  criteria  for  technical  order  managers. 

Question  14  (Why  do  you  think  you  were  selected  for  this 
job?),  another  open  question,  was  also  an  atterr.it  to  find 
out  what  the  respondents  thought  best  qualified  them  for 
placement  In  their  jobs.  Question  15  (Is  there  anything 
that  you  would  like  to  add  that  might  help  improve  the 
training  of  technical  order  managers?  If  so,  please 
elaborate. )  was  also  a  "catch-all”  attempt  to  prompt  the 
respondents  to  comment  further  on  any  other  ideas  they  might 
have  on  improving  the  present  process.  Finally,  question  16 
was  the  wrap-up  (What  do  you  think  should  be  included  in  the 
criteria  for  selection  of  technical  order  managers?).  This 
is  a  critical  question  -  what  the  people  who  are  actually  in 
the  job  think  others  should  possess  in  terms  of  skills, 
training,  experience,  knowledge,  etc.,  to  be  truly 
effective . 

The  validity  of  the  Instrument  should  be  evident  in  the 
selection  of  respondents.  Current  technical  order  managers 
were  asked  what  is  wrong  with  the  current  method  of 
selecting  and  training  technical  order  managers  and  what 
should  or  could  be  done  to  improve  it.  In  addition  to  their 
opinions,  a  variety  of  contractor  technical  order  (or 
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publications)  managers  were  asked  what  works  for  them. 

Their  responses  provided  additional  ideas  for 
recommendations  to  improve  the  current  method. 

Mr.  Chuck  Weaver  was  contacted  and  asked  to  review  the 
present  survey  and  offer  his  expert  opinion.  He  made  many 
valid  points  about  wording  and  terminology  and  suggested 
adjustments  in  order  to  produce  a  more  valid  final 
Instrument . 

A  preliminary  survey  was  administered  to  the  technical 
order  managers  in  one  of  Mr.  Art  Mungula's  SYS  230  classes. 
There  were  11  respondents  who  helped  out  immensely  in 
Identifying  non-mutually  exclusive  answers,  insufficient 
answer  options,  and  other  general  lack  of  clarity  areas. 

Sample/Population 

The  population  for  the  Air  Force  technical  order 
managers  Included  all  known  technical  order  managers  in  ASD. 
This  population  was  taken  from  a  list  maintained  by 
ASD/ALXS .  The  fact  that  a  more  official  list  is  not 
available  from  the  personnel  office  is  indicative  of  the 
selection  criteria  inadequacies  and  will  be  dealt  with  in 
the  recommendations  section  of  this  thesis. 

There  were  84  technical  order  managers  on  the  list  and 
every  manager  was  sent  a  questionnaire.  As  stated  earlier, 
only  ASD  was  surveyed  because  the  intention  of  this  research 
is  to  make  specific  recommendations  for  the  establishment  at 
ASD  of  selection  criterion  and  training  requirements  for 
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technical  order  managers.  The  data  as  initially  presented 
in  this  study  will  not  be  statistically  general izable  to  any 
population  other  than  ASD  technical  order  managers.  It  is 
felt  that  the  proposed  recommendations  can  best  be 
implemented  on  a  trial  basis  at  the  Division  level  rather 
than  at  the  MAJCOM  (Major  Command)  or  Air  Force  levels.  If 
the  recommendations  are  well  received  and  lead  to  successes, 
then  additional  steps  should  be  taken  to  further  survey  and 
generalize  the  data  to  encompass  the  rest  of  the  Command  and 
Air  Force  areas  where  technical  orders  are  managed. 

The  contractor  technical  order  sample  was  a  convenience 
sample  based  on  contractors  known  to  have  successful 
technical  order  management  practices.  The  convenience 
sample  method  can  be  justified  because  the  purpose  of 
surveying  the  contractors  is  not  to  obtain  a  statistically 
general izable  sample  but  to  obtain  ideas  of  what  does  and 
does  not  work  for  non-government  agencies. 

The  contractor  sample  was  taken  from  an  1989  Aerospace 
Industries  Association  (AIA)  directory,  which  lists  the 
members  of  the  AIA  Service  Publications  (technical  orders 
are  service  publications)  Committee  of  which  there  are  96 
members  representing  33  different  companies.  The 
contractors  contacted  were  General  Dynamics  Corporation, 
General  Electric  Aircraft  Engines,  Grumman  Aircraft  Systems 
Division,  Martin  Marietta  Corporation,  McDonnell  Douglas 
Corporation  and  Westinghouse  Electric  Corporation.  The 
specific  people  contacted  were  recommended  by  the  thesis 
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advisor  based  on  his  knowledge  of  their  experience  and 
willingness  to  assist  in  this  research  project. 

Data  Collection 

The  84  ASD  technical  order  managers  were  mailed 
questionnaires  and  return  envelopes  through  base 
distribution  in  March  1990.  A  cover  letter  was  attached 
explaining  who  the  researcher  was  and  why  she  needed  their 
participation  -  they  were  the  experts  and  in  the  best 
position  to  report  on  the  situation.  (The  cover  letter  and 
the  questionnaire  can  be  found  in  Appendix  A. )  The 
respondents  were  ask^d  to  reply  within  five  working  days  of 
receipt.  This  wi J  so  stated  so  that  those  who  had  been  on 
leave  or  away  on  travel  would  not  miss  a  "deadline."  They 
were  also  told  that  their  individual  responses  would  not  be 
attributed  to  them  personally  -  their  anonymity  would  be 
pr  eserved . 

After  one  month  had  passed  and  the  responses  had 
stopped  trickling  in  at  just  under  a  50  percent  response 
rate,  a  reminder  letter  (Appendix  B)  was  sent  out  to  all  the 
respondents  telling  them  it  was  not  too  late  to  return  the 
questionnaire  and  how  they  could  get  another  one  if  they  had 
"misplaced"  their  original.  This  brought  the  response  rate 
up  to  64.3  percent  or  54  out  of  the  84  mailed. 

The  contractor  technical  order  managers  were 
interviewed  via  telephone.  They  were  contacted  using  the 
"WATS"  line  in  the  graduate  student  administrative  office. 
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The  researcher  introduced  herself,  explained  the  reason  for 
the  call  and  how  she  had  gotten  their  name,  and  then  asked 
if  they  had  time  to  answer  a  few  questions.  The  researcher 
then  asked  the  six  questions,  clarifying  differing 
terminology  (most  contractors  call  their  technical  order 
managers  technical  publications  or  tech  pubs  managers)  and 
probing  further  when  additional  topics  of  interest  were 
mentioned. 

The  variables  are: 

1.  The  technical  order  managers'  experience. 

2.  The  technical  order  managers'  responsibilities. 

3.  What  they  think  is  important  in  terms  of 
training . 

4.  What  they  think  is  Important  in  terms  of 
experience . 

An  assumption  of  this  approach  is  that  the  technical 
order  managers  are  in  the  best  position  to  evaluate  the  most 
essential  requirements  for  superior  job  performance.  It 
might  be  argued  that  their  supervisors  would  have  a 
differing  opinion. 

This  research  is  also  limited  in  that  since  only  ASD 
technical  order  managers  were  surveyed,  the  results  are  not 
general izable  to  the  population  of  technical  order  managers 
throughout  the  Air  Force  or  Department  of  Defense.  The 
contractor  interviews  were  accomplished  in  order  to  get  some 
ideas  of  how  others  (with  typically  more  experience  and 
greater  continuity)  approach  the  same  problem.  As  stated 
earlier,  the  sample  taken  was  simply  a  focused,  small  size. 
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hand  picked  convenience  sample  and  therefore,  not 
generalizable  to  the  contractor  population. 

Statistical  Tests 

Since  this  is  a  descriptive  study  and  not  an  attempt  to 
show  causality,  the  information  obtained  will  be  analyzed 
with  frequency  analyses.  For  example,  the  more  frequently 
chosen  "shortcomings"  in  other  technical  order  managers  will 
be  considered  an  area  of  needed  emphasis,  and,  likewise,  the 
more  frequently  selected  strong  points  of  other  technical 
order  managers  will  be  considered  a  present  training  and 
selection  strong  suit. 

Chapter  Summary 

This  chapter  desrribed  the  procedures  used  in  the 
descriptive  study  to  determine  the  identity  of  the  ASD 
TOMA's  in  terms  of  grade,  education  and  experience;  their 
responsibilities  in  terms  of  dollar  value  and  number  of  TO's 
managed  as  well  as  requirements  involving  negotiations,  data 
item  descriptions,  logistics  support  analysis,  etc.;  and  how 
the  Air  Force  can  better  train  them  to  do  their  jobs.  The 
research  method  was  justified,  the  instruments  used 
explained,  the  populations  were  defined  and  the  data 
collection  and  statistical  tests  were  addressed. 
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IV.  Findings  and  Analysis 


This  chapter  gives  a  brief  description  of  the  surveys, 
a  recording  of  the  findings  depicted  by  both  pie  charts  and 
numerically  (Appendix  B),  and  an  analysis  of  the  findings  in 
terms  of  the  original  questions. 

Description  of  Surveys 

There  were  two  surveys  conducted  in  this  research.  The 
first  instrument  used  was  the  six-page  questionnaire  sent 
out  to  all  ASD  technical  order  managers.  It  was  accompanied 
by  a  cover  letter  signed  by  the  researcher  explaining  the 
questionnaire  and  the  motivation  behind  it.  A  copy  of  this 
questionnaire  and  the  accompanying  cover  letter  are  in 
Appendix  A. 

The  second  survey  conducted  was  a  telephone  interview 
with  six  different  defense  contractors  all  of  which  are 
large  companies  with  substantial  dollar  contracts  with  DOD 
(Department  of  Defense)  for  major  systems  and  less  than 
major  systems  acquisition.  The  contractors  interviewed  were 
General  Dynamics  Corporation,  General  Electric  Aircraft 
Engines,  Grumman  Aircraft  Systems  Division,  Martin  Marietta 
Corporation,  McDonnell  Douglas  Corporation  and  Westinghouse 
Electric  Corporation.  A  copy  of  the  list  of  questions  used 
to  conduct  these  telephone  interviews  is  in  Appendix  D. 
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Questionnaire 

As  mentioned  above,  questionnaires  were  sent  to  each  of 
the  84  technical  order  managers  in  Aeronautical  Systems 
Division.  Fifty-four  were  completed  and  returned  to  the 
researcher  for  a  return  rate  of  64.3  percent.  The  following 
is  a  recording  of  the  findings  as  well  as  an  analysis  of 
those  findings  with  respect  to  the  research  objectives  for 
each  question. 

Questionnaire  Question  1:  What  is  your  grade? 

Figure  1  shows  the  detailed  responses  to  this  first 
question.  There  were  no  non-responses  or  "Other"  responses 
to  this  question  and  all  respondents  were  able  to  answer 
with  one  of  the  options  provided.  As  can  be  determined  from 


Figure  1.  Grade  Distribution  of  Responding  Technical  Order 
Managers 
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Figure  1,  the  largest  group  of  TO  managers  were  E-7's  at  26 
percent,  followed  by  E-6  at  22  percent,  and  GS-12's  at  19 
percent.  There  were  no  0-5 's  or  above  and  there  were  no  GS- 
10's  or  below. 

The  first  question  broken  out  by  the  categories 
officer,  civilian  and  enlisted,  has  the  following 
percentages  respectively:  10,  61,  and  29. 

Questionnaire  Question  2:  What  is  the  highest  level  of 
education  you  have  achieved? 

The  responses  to  the  second  question  are  depicted  in 
Figure  2.  The  largest  number  of  responses  was  "Some 
College"  with  37  percent.  An  "Associate's  Degree"  was  the 
second  most  popular  response  with  28  percent,  "High  School 
Diploma  or  the  Equivalent"  had  13  percent,  a  "Bachelor'3 


Figure  2.  Education  Levels  of  Responding  Technical  Order 


Managers 
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Degree”  had  9  percent/  and  "Some  Graduate  Work"  had  7 
percent  responding.  Only  2  percent  did  not  finish  high 
school  while  4  percent  had  a  master’s  degree.  There  were  no 
respondents  with  doctoral  degrees  and  no  "Other”  responses. 

In  summary/  the  second  question  indicates  that  most  of 
the  technical  order  managers  have  some  college  but  not  a 
bachelor's  degree  as  yet. 

Questionnaire  Question  3:  How  long  have  you  been 
working  as  a  technical  order  manager? 

The  third  question  illustrates  a  fairly  experienced 
pool  of  T.O.  managers  as  depicted  in  Figure  3.  Sixty-four 
percent  of  the  respondents  had  two  or  more  years  of 
experience.  The  second  largest  group  of  respondents 


Figure  3.  Prior  Work  Experience  as  a  Technical  Order 
Manager 
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had  between  one  and  two  years  of  experience  (20  percent). 

The  responses  to  the  "more  than  nine  months  but  not  more 
than  a  year"  and  "three  to  nine  months"  categories  were 
equal  at  7  percent  each.  The  less  than  three  months 
category  had  only  a  two  percent  response  rate. 

The  third  question  asked  about  the  amount  of  experience 
the  respondents  had  as  technical  order  managers. 
Surprisingly,  64  percent  had  more  than  two  years  of 
experience.  This  is  surprising  because  that  was  one  of  the 
concerns  in  Billingham  and  Klassen's  (1:79)  research  in 
1905.  They  found  that  almost  50  percent  of  the  technical 
order  managers  had  less  than  two  years  experience. 

According  to  this  sample,  most  of  the  technical  order 
managers  are  quite  experienced.  Though  experience  is 
apparently  high,  continuity  within  programs  might  be  another 
issue.  Although  a  majority  of  the  respondents  have 
accumulated  a  sufficient  amount  of  TOMA  experience,  it  might 
not  have  necessarily  occurred  at  their  present  jobs.  The 
issue  of  program  TOMA  continuity  was  not  addressed  in  this 
questionnaire . 

The  increased  amount  of  experience  might  be  explained 
by  the  slow  down  since  1985  in  new  program  initiations  as 
well  as  the  slow  down  in  additional  SPO  personnel  to  support 
these  new  programs.  Or  perhaps  it  can  be  attributed  to  the 
decrease  in  PCS  (permanent  change  of  station)  moves  for 
military  (in  an  effort  to  save  funds)  leading  to  a  more 
stable  work  force. 


33 


Questionnaire  Question  4:  What  is  the  total  dollar 
value  of  the  technical  orders  you  manage? 

Figure  4  depicts  the  responses  to  the  fourth  question. 
The  largest  number  of  respondents  were  in  the  greater  than 
$25  million  category,  followed  by  the  $1  million  to 
$4,999,999  category  with  25  percent  and  then  the  "don't 
know"  category  which  had  13  percent.  The  $5  million  to 
$24,999,999  category  had  an  11  percent  response  rate,  and 
the  $250,000  to  $999,999  category  had  an  eight  percent 
response  rate.  There  was  one  response  that  said  the  amount 
was  classified.  This  response  was  not  listed  as  an  option 
on  the  questionnaire. 

The  dollar  amount  of  technical  orders  managed  is  useful 
information  in  that  this  kind  of  fiscal  responsibility 


CLASSFED  (l.9*)“i 
DOtfT  KNOW  (13.255) 


r-$25QK-$999K  (7.5%) 


>=  $25  ML  (41.5%) 


» — $1M— $4.®99M  (24.5%) 


$5M-$24.999M  (11.3%) 


Figure  4.  Total  Dollar  Value  of  Technical  Orders  Managed 
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should  get  the  attention  of  those  that  are  In  a  position  to 
emphasize  the  role  and  Importance  of  the  technical  order 
managers.  The  fact  that  42  percent  of  the  respondents  each 
manage  technical  orders  valued  at  equal  to  or  greater  than 
$25  million  is  impressive.  This  information  alone  certainly 
lends  them  greater  credibility. 

Questionnaire  Question  5:  Approximately  how  many 
technical  orders  do  you  manage? 

The  fifth  question  was  answered  as  depicted  in  Figure 
5.  The  largest  number  of  respondents  said  they  managed 
between  one  and  100  TO’s  (37  percent),  followed  by  101  to 
500  (20  percent)  and  then  the  ”don't  knows”  were  at  12 
percent.  The  501  to  1000,  the  1001  to  2000,  and  the  2001- 
3000  categories  each  had  a  nine  percent  response  rate.  The 


Figure  5.  Number  of  Technical  Orders  Managed 
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3001  to  5000  category  had  four  percent  responding.  No  one 

selected  the  "more  than  5000"  category. 

The  number  of  technical  orders  managed  is  not  really  a 

meaningful  or  useful  value  outside  of  their  Individual 

contexts.  As  one  of  the  respondents  pointed  out. 

The  number  of  T.O.'s  managed  is  not  Indicative  of 
actual  workload.  You  may  have  2000  TO's  and  be  a 
stable  program,  i.e.,  very  few  new/change  pages  going 
through  your  production  system,  or  it  could  be  very 
fluid. 

So  this  question  did  not  really  generate  any  additional 
useful  information. 

Questionnaire  Question  6:  In  what  capacity  have  you 
worked  with  tech  orders  prior  to  your  present  job  (flight 
line,  maintenance,  ln-shop,  in  the  System  Program  Office, 
etc.  )? 

Question  six  was  an  open  ended  question,  i.e.,  there 
were  no  answer  options  listed,  the  respondents  had  to 
generate  their  own  answers.  The  responses  were  grouped 
into  similar  categories  and  the  left  over  responses  were 
thrown  into  the  miscellaneous  category  as  illustrated  in 
Figure  6.  Most  of  the  responses  fell  into  the  "flight  line" 
category  with  67  percent  of  the  total  answers  (36  out  of  the 
54  respondents)  falling  in  this  grouping.  The  next  largest 
group  of  responses  was  the  "in-shop"  with  59  percent  (or  32 
out  of  54).  The  miscellaneous  category  was  the  third 
largest  at  28  percent  (or  15  out  of  54).  This  category 
included  such  responses  as: 

-  MAJCOM  functional  manager 

-  Air  Force  One  Program 

-  maintained  small  file  of  TO's 

-  used  TO's  on  the  job 
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Figure  6.  Areas  of  Prior  Experience  Working  with  Technical 
Orders 


-  AFOTEC  I  CBM  Test  Team 

-  posting  and  ordering 

-  development 

-  IG  Team 

-  TO  distribution  office 

(Whenever  responses  are  listed  in  this  thesis,  the  actual 
words  of  the  respondents  are  used  and/or  summarized  whenever 
possible  unless  otherwise  indicated.)  The  other  more  common 
responses  were  System  Program  Office  (11  percent),  none  (9 
percent),  validation  and  verification  (7  percent),  staff  (6 
percent)  and  training  (6  percent). 

Validation  and  verification  (val/ver)  is  a  procedure 
where  the  TOMA,  the  contractor  and  the  user  ensure  the 
technical  order  is  clearly  and  properly  written  and 
illustrated.  The  three  parties  actually  witness  a 
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technician  using  the  technical  order  and  verify  that  the 
desired  task  was  properly  accomplished. 

In  question  6  it  should  be  noted  that  the  answer 
categories  overlap,  that  is,  the  respondents  could  pick  more 
than  one  response,  whereas,  in  the  first  five  questions, 
only  one  response  could  be  chosen.  This  means  that  in  order 
to  get  the  percentages  in  the  pie  to  add  up  to  100,  the 
percentages  shown  in  Figure  6  must  be  based  on  the  total 
number  of  responses  and  not  the  total  number  of  respondents. 
There  were  54  respondents,  but  many  wrote  more  than  one 
answer  so  the  total  number  of  answers  recorded  and 
categorized  was  104.  For  example,  36  of  the  54  respondents 
cited  flight  line  experience  which  is  a  67  percent  rate. 

But  there  were  104  total  answers  and  36  out  of  104  is  35 
percent  which  is  what  Figure  6  indicates.  It  was  felt  that 
the  most  useful  part  of  the  response  analyses  would  be  the 
pie  charts  because  of  the  ease  with  which  one  can  determine 
the  most  frequent  responses  just  by  quickly  glancing  at 
them.  So,  although  the  percentages  would  be  different  if 
based  on  the  number  of  respondents  rather  than  the  number  of 
responses,  when  compiled  into  a  pie  chart,  the  slice  of  pie 
would  be  the  same  relative  size  in  either  case. 

Obviously,  the  most  frequent  work  experience/ 
backgrounds  for  the  ASD  technical  order  managers  is  in  the 
area  of  ln-shop  and  flight  line  maintenance  as  determined 
from  the  responses  to  question  6.  This  viewpoint  should 
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also  be  remembered  when  reviewing  the  backgrounds  for  other 
technical  order  managers  as  suggested  by  these  TOMA's. 

Questionnaire  Question  7:  What  kind  of  technical  order 
acquisition  manager-related  training  have  you  had? 

The  responses  to  the  seventh  question  are  depicted  in 
Figure  7.  The  most  popular  training  course  was  SYS  230  Air 
Force  Technical  Order  Acquisition  and  Management  (37 
percent),  followed  by  SYS  100  Introduction  to  Acquisition 
Management  (34  percent),  then  AFALC  001  Deputy  Program 
Manager  for  Logistics  (DPML)  Course  (16  percent),  SYS  225 
Acquisition  Logistics  (6  percent).  Systems  028  Introduction 
to  Configuration  Management  (6  percent),  and  none  (1 
percent).  As  in  question  6,  the  answer  categories  to  thl3 
question  overlap,  i.e.,  more  than  one  answer  could  be  given. 
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Question  7  responses  indicated  that  83  percent  (or  45 
out  of  the  54  respondents)  have  taken  SYS  230  Air  Force 
Technical  Order  Acquisition  and  Management.  This  is  an 
encouraging  ratio  considering  that  only  64  percent  of  the 
respondents  have  worked  as  a  TOMA  for  two  years  or  more. 
Those  that  had  not  taken  the  class  vociferously  expressed 
concern  that  TOMA’s  are  not  given  priority  over  non-TOMA's 
in  the  class  attendance.  The  SYS  10.0  Introduction  to 
Acquisition  Management  percent  was  quite  high  also  at  76 
percent.  AFALC  001  the  DPML  course  had  quite  a  high 
attendance  rate  at  35  percent. 

Questionnaire  Question  8:  What  type  of  experience  did 
you  have  to  qualify  you  for  this  job? 

Question  8  responses  are  displayed  in  Figure  8.  The 


OTHER  (13.3%)- 
NONE  (2.7%)- 


DC  REVEWS  (6.2%)- 


HANDS  ON  (34.5%) 


SOWs  (8.8%) 

NEGOTIATIONS  (5.3%) 

CUD  CONF  (13.3%) 


PR's  (15.9%) 


Figure  8.  Technical  Order-Related  Experience  Qualifying  the 
Technical  Order  Managers  for  Their  Jobs 
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most  common  type  of  experience  was  "hands-on"  (72  percent), 
followed  by  "in-process  reviews"  (33  percent),  guidance 
conferences  (28  percent),  and  the  "other"  category  (28 
percent)  which  included: 

-  verification/validation 

-  prepublication  reviews 

-  inspection  team 

-  avionics  Instructor-  staff 

-  no  elaboration 

-  maintenance  background,  common  sense 

-  DCAS  -  electronics  QAS  (Quality  Assurance) 

-  TO  specif ications  and  standards 

review/tailor ing/OPR 

-  Writing  data  item  descriptions 

-  TMCR-86-01  author  (Tech  Manual  Contract  Requirement) 

-  TO  acquisition  policy 

-  source  selection 

-  some  LSA  review 

-  represented  TOMA  at  meetings,  conferences  etc. 

-  training  program  selectee  based  on  merit  in  previous 

jobs  and  education  level 

The  last  four  categories  were  statements  of  work  (18 
percent),  data  item  description  review  (13  percent), 
negotiations  (11  percent),  and  none  (6  percent).  Again,  as 
explained  in  question  6,  percentages  in  Figure  8  are  based 
on  total  number  of  responses  while  the  percentages  listed 
above  are  based  on  the  total  number  of  respondents. 

The  eighth  question  was  quite  a  subjective  one  in  that 
it  asked  the  TO  managers  to  list  what  experience  they 
thought  had  qualified  them  for  the  jobs  they  now  held. 
Seventy-two  percent  said  their  hands-on  experience  had 
qualified  them.  This  is  expected  because  most  of  the  ASD  TO 
managers  are  hired  because  they  have  hands-on  experience 
using  technical  manuals/orders.  Other  large  categories  were 
the  in-process  reviews  and  the  guidance  conferences  at  33 
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and  28  percents  respectively.  These  answers  are  significant 
because  they  reflect  what  the  respondents  feel  are  areas 
related  to  what  they  know  now,  i.e.,  what  experience  most 
qualified  and  hence  helps  them  now  in  the  performance  of 
their  jobs.  So,  in  essence,  what  the  respondents  are  saying 
here  is  that  hands-on  knowledge  of  technical  orders,  as  well 
as  in-process  review  and  guidance  conference  experience  are 
essential  to  being  a  technical  order  manager. 

Questionnaire  Question  9:  What  kind  of  experience, 
prior  to  your  coming  on  the  job,  would  have  better  prepared 
you  for  the  job  you  have  now? 

The  ninth  question  had  a  fairly  equal  distribution  of 
responses  as  indicated  in  Figure  9.  The  most  common 
response  was  negotiations  (44  percent  -  based  on  the  number 


Figure  9.  Prior  Experience  that  Would  Have  Better  Prepared 
the  Technical  Order  Managers 
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of  respondents  -  see  question  6  for  further  explanation), 
followed  by  guidance  conferences  (43  percent),  ptatrr.  ts  of 
work  (41  percent),  in-process  reviews  (39  percent),  and 
other  (35  percent)  which  included: 

-  Attendance  in  Systems  230 

-  participate  in  using  command’s  preparation  of 
Statement  of  Need 

-  management  experience,  administrative  experience 
(letter  generating,  coordination,  etc),  experience  in 
contracts,  logistics,  dealing  with  contractors, 
conducting  meetings 

-  complete  all  training  courses  offered  at  AFIT 

-  training  on  how  to  understand  contracts-  experience 
in  TO  preparation  format  and  some  of  the  more  important 
Mil-Spec  guidelines  that  are  used 

-  familiarization  with  specs  used  in  the  TO  world, 
different  forms  being  used,  concepts  and  policies, 
i.e.,  overview  of  what  TO  acquisition  is  about 

-  proposal  cost  review  analysis 

-  program  manager,  contracting,  manufacturing/ 
production  systems,  software  engineering 

-  quality  assurance  experience 

-  running  a  large  day-care  center  for  3-5  year  olds, 
this  would  have  given  me  more  insight  in  how  to  deal 
with  the  various  Air  Logistics  Centers 

The  next  most  frequently  chosen  category  was  data  item 

description  reviews  (33  percent),  hands-on  experience  (26 

percent)  and  no  response,  i.e.,  nothing  was  selected  or 

written  in  (3  percent). 

The  last  "other”  comment  listed  above  needs  to  be 
addressed.  Since  the  researcher  has  been  on  both  sides  of 
the  AFLC/AFSC  wall,  an  experienced,  albeit  it  subjective, 
explanation  for  the  comment  will  be  suggested.  The  Systems 
Command  TOMA  is  expressing  an  often  felt  frustration  about 
the  seemingly  Impossible  task  of  satisfying  the  Air  Logistic 
Command  (ALC)  personnel.  They  (the  ALC’s)  always  want  what 
they  do  not  need  or  more  than  what  they  do  need  and  they 
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take  it  very  personally  when  they  cannot  get  what  they  want. 
(This  is  from  the  frustrated  System  Command  TOMA's 
viewpoint.)  Now  the  other  side  of  the  coin  is  the  ALC 
person's  view.  This  person  is  frustrated  because  Systems 
Command  insists  on  spending  money  on  things  they  do  not  want 
and  refuses  to  purchase  things  they  feel  they  need  for  the 
follow-on  support  of  the  weapon  system  for  which  the  ALC  is 
responsible.  No  matter  how  stringently  the  ALC  lobbies  for 
its  way,  the  SPO  can  still  choose  to  ignore  their  desires 
prior  to  PMRT  because  the  SPO  has  the  final  word  since  they 
control  the  funding.  This  situation  has  often  proven 
Incredibly  frustrating  to  the  ALC  personnel  because  they 
often  do  not  feel  that  the  SPO  has  their  (the  ALC's)  best 
interests  at  heart.  This  is  because  after  PMRT  the  SPO  will 
largely  disband  except  for  those  working  the  residual  tasks. 
The  SPO  personnel  will  then  move  to  other  SPO's  and  not  have 
to  contend  with  any  of  their  decisions,  while  the  ALC  people 
are  left  holding  the  "bag."  This  so-called  "bag"  contains 
contracts  and  agreements  that  the  ALC  never  wanted  in  the 
first  place  but  that  they  now  have  to  live  with.  After 
bumping  heads  unsuccessfully  with  several  SPO  personnel  whom 
the  ALC  employee  did  not  feel  had  their  best  interests  at 
heart  (or  perhaps  did  not  even  consider  them  in  their 
decisions),  the  ALC  personnel  could  start  to  get  a  bit 
frustrated,  resentful,  and  yes,  even  childish  when  dealing 
with  their  "inconsiderate"  SPO  counterparts.  Unless  there 
is  good  leadership  at  the  SPO  Director  and  DPML  level  as 
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well  as  the  ALC  Program  Manager  level  to  set  the  example  and 
the  climate,  the  above  scenario  will  continue  adding  to 
personnel  frustrations  and  program  costs. 

Question  9  also  gets  more  into  the  central  issue  of 
this  thesis  and  again,  as  in  question  8,  it  asks  a 
subjective  question.  The  respondents  were  asked  to  circle 
what  areas  of  prior  experience  would  have  better  prepared 
them  for  their  present  jobs.  This  question  is  very 
significant  in  that  it  should  point  to  the  areas  that  are 
presently  lacking  as  far  as  selecting  prepared  people  to  be 
technical  order  managers.  Forty-four  percent  of  the 
respondents  listed  negotiations  as  an  area  of  experience 
that  would  have  better  prepared  them.  A  drawback  of  this 
response  is  that  "negotiations**  is  never  specifically 
defined.  Legally,  only  certified  contract  negotiators  are 
allowed  to  negotiate  with  the  contractor.  So  why  do  so  many 
TOMA's  want  more  training  in  negotiations?  The  researcher 
proposes  that  because  the  contracting  officer  must  come  to 
the  TOMA  for  cost,  time  required,  and  other  data 
requirements  in  order  to  prepare  for  negotiations,  the  TOMA 
is  often  asked  to  contribute  and  participate  in  an  area  of 
expertise  the  TOMA  knows  very  little  about.  The  contracting 
officer  does  not  have  time  to  train  every  TOMA  in  the 
essentials  of  negotiating  techniques  but  a  basic 
understanding  of  such  would  help  the  TOMA's  immensely  in 
their  collaborations  with  the  contracting  officer. 
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The  hands-on  choice  was  relatively  infrequently  chosen 
•at  26  percent.  This  corresponds  to  the  responses  to 
question  8  where  72  percent  said  they  were  qualified  for 
their  jobs  because  of  their  hands-on  experience.  In  other 
words,  those  that  were  not  chosen  on  the  basis  of  their 
hands-on  experience  feel  that  experience  would  have  better 
prepared  them  for  their  jobs.  Unfortunately,  not  many  of 
those  with  the  hands-on  experience  have  had  an  opportunity 
to  get  versed  in  the  more  SPO-oriented  activities  such  as 
those  listed  here  —  negotiations,  guidance  conferences, 
etc . 

This  question  could  have  been  slightly  slanted  in  that 
there  were  six  provided  answers.  The  answers  were  a  best 
attempt  to  list  the  full  range  of  technical  order  manager- 
related  duties  and  not  just  the  areas  that  the  researcher 
thought  might  be  lacking,  i.e.,  "hands-on  experience"  (which 
the  researcher  felt  was  an  area  in  which  most  of  the  TOMA's 
were  experienced)  was  listed  as  well  as  "negotiations"  which 
was  an  area  in  which  the  researcher  thought  the  TOMA's  might 
feel  inadequate.  There  was  also  an  "other"  answer  option 
and  it  was  chosen  by  35  percent  of  the  respondents.  Because 
of  the  variety  of  responses  given  in  the  "other"  category, 
it  is  not  felt  that  the  respondents  were  "guided"  into  their 
answers  on  this  or  any  other  question. 

Questionnaire  Question  10:  What  kind  of  training  would 
you  still  like  to  have  in  order  to  improve  your  present 
level  of  job  performance? 
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The  tenth  question  was  responded  to  as  indicated  in 
Figure  10.  The  most  frequently  chosen  response  was  SYS  225 
Acquisition  Logistics  (65  percent),  followed  by  SYS  028 
Introduction  to  Configuration  Management  (44  percent),  SYS 
229  Test  and  Evaluation  (35  percent),  AFALC  001  Deputy 
Program  Manager  for  Logistics  ( DPML)  Course  (26  percent)  and 
"Other"  (28  percent).  (Again,  as  in  question  6,  the 
percentages  in  Figure  10  are  based  on  the  total  number  of 
responses  while  the  percentages  given  in  the  text  are  based 
on  the  number  of  respondents.)  The  "Other"  responses 
included : 

-  an  introductory  course  to  test  and  evaluation  (not 
necessarily  229) 

-  SYS  230  needs  to  be  changed  to  be  TOMA  friendly  and  a 


Figure  10.  Training  Desired  to  Improve  Present  Job 
Per f ormance 
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good  OJT  program  needs  to  be  made  and  used 

-  need  more  mil-spec  application  knowledge 

-  SYS  006  Intermediate  System  Acquisition 

-  following  a  training  plan  that  is  made  just  for 
technical  order  managers  along  with  a  797  for  the 
records 

-  technical  writing  classes  (for  evaluating  paper 
TO '  s ) ,  computer  programming  classes  (for  evaluating 
digital  TO's,  LOG  260  Provisioning  Mgmt,  LOG  224 
Logistics  Mgmt,  SYS  227  Financial  Mgmt  in  Weapon  System 
Acquisition,  SYS  200  Acquisition  Planning  and  Mgmt 

-  seminars  dealing  with  current  Issues  -  CALS/JUSTIS 
(Computer  Aided  Logistics  System/Joint  Uniform  Services 
Technical  Information  System),  distribution  statement, 
AFR  80-34  reguests/FOI A  (Freedom  of  Information  Act), 
i.e.,  protection  of  data,  LSA,  TM  86-01 

-  LOG  260,  SYS  370,  Provisioning  and  Data  Management 

-  SYS  200 

-  some  type  of  program  that  would  better  prepare  us  for 
tech  order  pricing 

-  LSA/LSAR  (Logistics  Support  Analysis  Requirements) 
training  -  reason  -  the  TO's  and  nearly  all  the  other 
areas  of  logistics  originate  from  LSA  -  so  it's 
imperative  to  know  what  LSA  requirements  to  put  on 
contract  and  what  these  requirements  will  provide  us 

The  frequency  of  "other"  responses  indicates  a  sincere 

interest  in  improved  training.  The  comments  seem  to 

indicate  a  desire  for  more  practical  Information/training 

that  will  prepare  them  for  what  they  are  currently  doing  in 

their  jobs.  Words  like  "TOMA  friendly"  (meaning  perhaps, 

more  in  line  with  what  the  TOMA  "needs"  to  know  -  not  what 

is  "nice"  to  know),  CALS/JUSTIS,  FOIA,  LSA,  tech  order 

pricing,  all  seem  to  point  to  the  current,  and  likewise,  not 

well  understood,  issues.  Clearly,  the  present  training 

opportunities  should  be  looked  at  from  this  perspective. 

The  remaining  responses  included  SYS  230  Air  Force 

Technical  Order  Acquisition  and  Management  (20  percent),  SYS 

100  Introduction  to  Acquisition  Management  (13  percent),  and 

the  non-responses  which  constituted  only  seven  percent  of 
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the  total  answers.  The  percentages  of  TOMA's  who  want  to 
take  SYS  230  or  SYS  100  correspond  with  the  percentages  of 
those  who  have  already  taken  the  courses  as  determined  in 
question  7.  In  other  words,  76  percent  had  already  taken 
SYS  100  and  13  percent  still  wanted  to  take  it  and  likewise, 
83  percent  had  already  taken  SYS  230  and  20  percent  still 
wanted  to  take  it.  (Admittedly,  there  is  some  overlap  in 
the  SYS  230  percentages,  perhaps  attributable  to  those  who 
would  like  to  repeat  the  course.) 

Question  10  addressed  the  training  desired  by  the 
technical  order  managers  to  improve  their  present  job 
performance.  This  is  an  important  question  because  of  the 
higher  experience  level  of  responding  technical  order 
managers.  Based  on  the  level  of  experience  they  have  (most 
being  over  two  years)  what  kind  of  training  (or  in  what  area 
do  you  still  feel  weak)  would  you  still  like  to  have  in 
order  to  Improve  your  present  level  of  performance. 
Consistent  with  the  previous  answers,  most  of  the 
respondents  wanted  training  in  the  acquisition  logistics 
area  (SYS  225).  This  is  consistent  because  most  of  them 
have  backgrounds  on  the  flight  line  and  maintenance  shops 
and  are  weak  in  the  System  Program  Office/acquisition 
logistics  area.  SYS  028  (Introduction  to  Configuration 
Management)  and  SYS  229  (Test  and  Evaluation),  both  common 
SPO  domains,  were  also  frequently  chosen. 

Questionnaire  Question  11:  What  shortcomings  have  you 
witnessed  in  other  technical  order  managers? 
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The  eleventh  question  was  responded  to  as  indicated  in 
Figure  11.  These  responses  were  quite  equitably  distributed 
with  lack  of  acquisition  background  getting  the  most 
responses  (72  percent  -  based  on  the  number  of  respondents  - 
see  question  6  for  further  explanation)  followed  by  lack  of 
negotiation  experience  (61  percent),  lack  of  cost  evaluation 
experience  (56  percent),  lack  of  data  item  description 
experience  (52  percent),  lack  of  statement  of  workexper ience 
(50  percent),  lack  of  tactfulness  in  dealing  with  people  (48 
percent),  lack  of  in-process  review  experience  (39  percent), 
lack  of  guidance  conference  experience  (39  percent),  lack  of 
configuration  background  (39  percent ), lack  of  knowledge  of 
technical  orders  (31  percent),  and  "other"  at  17  percent. 


Figure  11.  Shortcomings  Witnessed  In  Other  Technical  Order 
Managers 
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Some  of  the  other  responses  included: 

-  lack  of  training/exper ienae  in  dealing  with 
bureaucracy  and  matrix  mgmt  in  SPO 

-  lack  of  overall  experience  in  everyday  happening  in 
military  such  as  how  to  conduct  yourself  in  the 
presence  of  senior  service  personnel  and  aircraft 
industry  mgmt 

-  PMRT  planning 

-  lack  of  ability  to  make  a  decision  -  "You  can  always 
count  on  SPO  TOMA  for  a  definite  maybe!" 

Five  of  the  nine  "other"  responses  did  not  elaborate.  The 

only  remaining  responses  were  in  the  "no-response"  category 

of  which  there  was  only  4  percent. 

Question  11  is  attempting  to  find  out  in  what  areas  the 

respondents  have  witnessed  other  technical  order  managers' 

weaknesses.  This  question  assumes  the  TOMA's  are 

knowledgeable  enough  to  recognize  shortcomings  of  their  own. 

The  weakness  of  the  question  might  be  found  in  that  the 

respondents  are  most  likely  to  recognize  shortcomings  only 

in  areas  that  they  are  knowledgeable  in  themselves,  i.e., 

they  might  not  recognize  an  inadequate  knowledge  of 

configuration  management  if  they  are  weak  in  that  area. 

Consequently,  an  important  category  of  knowledge  could 

conceivably  be  overlooked  because  not  enough  TOMA's  have  the 

experience  to  recognize  the  shortcoming.  However,  this  is 

not  considered  a  major  issue  because  of  the  high  percentage 

of  experience  of  the  respondents.  Another  possible  weakness 

of  this  question  is  that  the  frequency  of  selection  of  the 

shortcomings  might  not  directly  correspond  to  the  actual 

frequency  of  occurrence.  Possibly,  those  who  cited  the  lack 

of  cost  evaluation  knowledge  as  a  shortcoming,  witnessed 
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this  shortcoming  in  the  same  person  so  the  frequency  is 
misrepresented  by  the  nymber  of  witnesses  who  saw  the  same 
person  rather  than  the  number  of  actual  TOMA's  with  this 
shortcoming.  As  mentioned  in  the  previous  paragraph,  the 
answers  are  fairly  evenly  distributed. 

The  largest  category  selected  in  question  11  is 
acquisition  background  with  72  percent,  again  consistent 
with  previous  findings.  The  majority  of  the  options,  as  the 
majority  of  the  TOMA's  duties,  had  to  do  with  the  SPO 
responsibilities.  The  only  option,  other  than  "other",  that 
was  not  exactly  SPO  related  was  the  "lack  of  knowledge  of 
technical  orders".  The  percentage  citing  this  shortcoming 
was  relatively  low  but  not  insignificant  at  31  percent. 

Questionnaire  Question  12:  What  strong  points  have  you 
witnessed  in  other  tech  order  managers? 

The  responses  to  the  twelfth  question  are  portrayed  in 
Figure  12.  The  most  frequently  chosen  answer  was  knowledge 
of  technical  orders  (67  percent  -  based  on  the  number  of 
respondents  -  see  question  6  for  further  explanation), 
followed  by  tactfulness  in  dealing  with  people  (59  percent), 
acquisition  background  (43  percent),  in-process  review 
experience  (43  percent),  guidance  conference  experience  (37 
percent),  statement  of  work  experience  (33  percent),  cost 
evaluation  experience,  configuration  experience,  knowledge 
of  negotiation  techniques,  and  data  item  description 
experience,  all  at  30  percent  each.  The  least  often  chosen 
category  was  the  "other"  category  which  was  chosen  only  nine 
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Figure  12.  Strong  Points  Witnessed  in  Other  Technical  Order 
Managers 

percent  of  the  time.  Three  of  these  "other"  choosers  did 
not  elaborate.  Elaboration  on  the  "other"  choice 
includedgood  technical  background  and  effectively  managing 
within  the  bureaucracy  and  matrix  management  of  the  SPO. 

This  question  is  an  attempt  to  determine  what  the 
technical  order  managers  feel  are  important  strong  points  to 
have  as  well  as  how  many  other  TOMA’s  have  they  witnessed 
with  these  strong  points.  In  other  words,  in  what  areas  are 
we  doing  well?  (There  are  the  same  possible  weaknesses  and 
assumptions  in  this  question  as  cited  in  question  11  above.) 
Again,  technical  order  experience  and  acquisition  background 
come  out  strong  as  being  Important  to  the  successful 
technical  order  manager.  A  rather  surprising  strong  showing 
came  from  "tactfulness  in  dealing  with  others."  This 
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characteristic  is  obviously  an  important  element  of  the 
successful  technical  order  manager  yet  it  is  often 
overlooked  while  the  more  technical  and  quantifiable 
features  such  as  years  of  experience  and  training  completed 
are  concentrated  on. 

Questionnaire  Question  13:  Please  try  to  list  and 
explain,  if  necessary,  all  of  your  duties  associated  with 
technical  order  management. 

Question  13  was  an  open-ended  question  that  was  trying 
to  determine  the  mix  of  duties  assigned  to  most  ASD  TOMA's 
and  to  also  see  if  any  large  categories  were  possibly 
excluded  in  the  list  of  choices  to  the  previous  questions. 
There  were  three  examples  of  answers  given  in  the 
questionnaire:  negotiations,  in-process-review3,  and  data 

item  description  review.  These  three  suggestions  were 
respectively  the  fourth,  first  and  fifth  most  often  chosen 
answers  (as  indicated  in  Figure  13)  so  it  is  not  felt  that 
the  suggestions  unfairly  biased  the  answers  (as  might  have 
been  the  case  if  the  answers  were  the  three  most  often 
chosen ) . 

Again,  as  in  question  6,  more  than  one  answer  could  be 
given  by  each  respondent,  consequently,  the  percentages 
referred  to  in  the  text  are  different  from  those  in  the 
chart  because  the  chart  percentages  are  based  on 
totalanswers  given  versus  the  text  percentages  which  are 
based  on  the  number  of  respondents. 

The  most  common  duty  listed  was  in-process-reviews  (65 
percent),  followed  by  guidance  conferences  (44  percent). 
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DID  (13.5%) 


Figure  13.  The  Technical  Order  Managers'  Duties 


statements  of  work  (41  percent),  validation/verification  (39 
percent),  negotiation  and  data  Item  description  (both  35 
percent),  cost  evaluation  (19  percent),  pre/post  publication 
reviews  (17  percent),  and  no  response  (13  percent).  There 
was  a  large  number  that  fell  into  the  "other"  category  and 
because  of  its  size  and  variation,  this  category  was  not 
placed  on  the  pie  chart  in  Figure  13.  The  following  is  a 
list  of  the  "other"  responses  that  were  given  by  more  than 
one  person: 

-  review  ECPs/CCPs/ACSN 

-  SERDS 

-  TO  specifications  and  standards 

-  TMCR  86-01  author/tailor 

-  TO  acquisition  policy 

-  technical  interchange  policy 

-  program  management 
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-  ensure  contractor  complies  with  specs 

-  scheduling 

-  data  call  input 

-  fact  find 

-  LSAR  reviews 

-  inputs  to  PMRT  planning 

-  Form  27,  AFTO  22's 

-  TO  revisions/updates/changes 

-  personnel  mgmt 

-  briefings/preparation 

-  FMS  CSTO's 

-  publication/delivery  of  TO's 

-  CDRL  deliveries/packages 

-  source  selections 

-  budgeting/funding 

-  requirements  identification 

-  support  equip,  coordination/acquisition 

-  CFAEs/CFEs 

-  "everything" 

-  TO  review  boards 

This  question  uncovered  some  responses  that  had  not 
been  listed  as  options  in  previous  questions:  validation/ 
verification  (val/ver)  and  the  pre/post  publications  review 
tasks.  The  val/ver  was  especially  significant  because  it 
was  cited  by  39  percent  of  the  respondents.  With  the 
exception  of  the  ?<hands-on"  experience,  the  answers  to  this 
question  correspond  to  those  in  question  9  which  asked  what 
kind  of  prior  experience  would  have  better  prepared  the 
TOMA's  for  their  present  jobs  (see  Figure  9).  Both  sets  of 
responses  are  rather  evenly  distributed  and  the  slightly 
larger  groups  Include  ln-process-reviews,  negotiations, 
guidance  conferences,  statements  of  work  and  data  item 
descriptions.  The  only  area  left  out  of  Figure  9  is 
val/ver,  and  this  is  possibly  because  val/ver  was  not  listed 
as  an  option  in  question  9.  The  point  of  highlighting  the 
relationship  between  the  two  questions  is  that  the 
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respondents  are  saying  that  they  need  more  experience  in 
almost  every  area  in  which  they  work  as  technical  order 
managers  (the  exceptions  being  cost  evaluation  and  val/ver). 
Obviously,  there  is  room  for  improvement  in  the  training 
area. 

Questionnaire  Question  14:  Why  do  you  think  you  were 
selected  for  this  job? 

Question  14  elicited  some  varied  responses  as  indicated 
in  Figure  14.  The  most  common  response  was  that  the 
selection  was  based  on  the  individual's  experience  (30 
percent  -  based  on  number  of  respondents  -  see  question  6 
for  further  explanation),  followed  by  the  response  that  the 
individual  was  available  or  they  volunteered  (20  percent), 
or  based  on  their  maintenance  background  (15  percent),  or 


Figure  14.  Why  the  Technical  Order  Managers  Were  Selected 


for  Their  Jobs 
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based  on  a  PCS  (permanent  change  of  station)  move,  luck  or 
staff  experience  (7  percent  each),  or  based  on  previous 
acquisition  jobs  held  (6  percent).  There  were  quite  a 
number  of  varied  "other"  responses  that  were  not  included  in 
the  pie  chart  in  Figure  14  because  their  volume  would  have 
dwarfed  the  other  slices.  The  "other"  responses  included 
the  following  (the  numbers  in  parentheses  after  a  responses 
Indicate  frequency  of  occurrence  if  greater  than  one). 

-  common  sense 

-  willingness  to  work  (2) 

-  most  qualified  within  the  program 

-  aircraft  background  (2) 

-  tech  order  experience  (2) 

-  management  experience 

-  old  boy  network 

-  good  record  (2) 

-  humanitarian  assignment 

-  best  trainee  candidate  at  the  time 

-  varied  weapon  system  and  aircraft  background 

-  completed  SYS  230 

-  previous  civil  service  experience  (DCAS) 

-  verification  team  member  and  manager 

-  SPO  experience 

-  F-16  avionics  system  experience 

-  most  qualified 

-  don't  know 

-  no  response  (2) 

Question  14  is  a  subjective  one  in  that  the  respondents 
are  being  asked  to  speculate  on  why  they  were  chosen  for 
their  jobs.  The  majority  of  their  answers  indicated  either 
maintenance  background  or  some  type  of  experience.  Only  a 
small  percentage  indicated  luck,  PCS,  availability,  etc., 
which  is  encouraging  because  one  would  like  to  think  that 
most  of  these  critical  positions  are  filled  with  qualified 
people  and  not  just  available  ones.  The  limitation  of  this 
question  is  that  it  asks  those  getting  moved  around  and  it 
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does  not  get  the  viewpoint  of  the  people  who  do  the  moving. 
The  commanders  of  product  divisions,  SPO  directors,  program 
managers  and  military  and  civilian  personnel  people  might 
have  a  different  view  of  why  the  respondents  are  where  they 
are  other  than  "luck."  And  such  responses  as  "common  sense" 
and  "willingness  to  work"  are  probably  not  reasons,  in  and 
of  themselves,  that  someone  was  chosen  as  a  TOMA.  As  the 
system  for  selecting  TOMA's  improves,  one  would  probably  see 
an  Improvement  in  the  responses  to  this  question  (an 
improvement  being  defined  as  more  respondents  saying  they 
were  chosen  because  of  their  background  and  experience  and 
less  saying  they  were  chosen  because  of  luck  and  the  fact 
that  they  were  available). 

Questionnaire  Question  15:  Is  there  anything  that  you 
would  like  to  add  that  might  help  improve  the  training  of 
tech  order  managers?  If  so  please  elaborate. 

Question  15  was  open  ended  and  produced  such  a  variety 
of  responses  that  it  was  not  feasible  to  group  the  responses 
into  a  pie  chart.  The  large  majority  of  the  responses  were 
well  thought  out  and  positively  intentioned.  Obviously,  the 
TOMA's  took  the  question  seriously  and  made  some  very 
positive  comments.  The  following  is  a  summary  of  those 
comments,  grouped  by  the  general  category  into  which  they 
fell.  Asterisks  have  been  placed  beside  the  more  pivotal 
issues  raised  which  are  addressed  following  the  list. 

(Though  most  of  the  comments  are  summarized,  an  attempt  was 
made  to  use  the  respondents'  own  words  as  much  as  possible 
and  to  avoid  interpretation  where  the  meaning  was  not  clear. 
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Hence,  the  incomplete  sentences  and  occasionally  incorrect 
English. ) 

Training 

-  every  TO  manager  should  have  to  take  SYS  230  early  on 
(3  comments  on  this) 

-  training  needs  to  be  more  extensive  -  include  all 
aspects  of  tech  order  management  -  and  timely  (2) 

-  ASD/ALXS  has  developed  an  excellent  training  plan  but 
needs  more  emphasis  on  contract  evaluation  and 
negotiation 

-  need  cost  evaluation,  negotiation  and  SOW  preparation 
training  during  first  year  as  a  TOMA 

-  new  training  program  developed  by  ASD/ALXS  will  be 
great 

-  need  formal  training  plan  for  TOMA's 

-  need  training/emphasis  on  latest  reg  and  mil-spec 
changes 

*  -  need  more  practical  seminars  like  the  ones  ALXS  is 
currently  running  and  less  formal  AFIT  classroom 
training 

-  people  need  to  understand  the  software  and  equipment 
they're  buying  TO's  for  -  software  changes  to  OFP's  and 
support  equipment  have  a  significant  impact  on  TO 
changes 

-  need  training  up  front  -  not  several  years  into  the 
job  (3) 

-  need  training  prior  to  assignment  (2) 

-  send  personnel  to  acquisition  related  school  prior  to 
becoming  a  TOMA  -  SAS  006  Intermediate  System 
Acquisition  course  at  Brooks  AFB  provides  a  Systems 
Command  approach 

-  training  should  be  a  higher  priority  item  -  currently 
it's  hit-or-miss  -  often  TOMA's  can't  get  in  needed 
classes  because  they're  filled  with  GS-5  secretaries  et 
al.  who  will  never  use  the  info 

*  -  the  AFIT  courses  are  too  restricted  by  quotas  so  that 
the  enlisted  people  who  really  need  the  training  are 
bumped  out  by  the  professional  students 

-  training  needs  to  be  more  flexible  to  fit  the  toma- 
ln-training ' s  background  and  program,  e.g.,  a  program 
acquiring  only  digital  data  will  have  little  use  for 
training  in  plate  negative  inspections,  etc. 

*  -  SYS  230  still  didn't  tell  me  what  I  need  to  manage 
day-to-day  -  might  be  my  organization's  fault  but  I 
still  don't  know  what's  expected  of  me  and  what  I'm 
authorized  to  do 

*  -  need  courses  that  give  you  more  practical  experiences 
such  as  conducting  IPR's  and  guidance  conferences 

-  design  training  courses  for  specific  areas  of  TO 
acquisition,  i.e.,  munitions,  CFE/CFAE  (contractor 
furnished  equipment/aeronautical  equipment)  notices. 
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SERDs,  interface  between  the  ASD  product  division  and 
the  ALC's,  the  ALC's  capabilities  and  their  TO 
acquisition  processes,  also  their  TO  maintenance 
systems  tATOS  (Automatic  TO  System)! 

-  TOMA's  should  have  approximately  6  months  general 
experience  in  the  TO  acquisition  area,  then  be  sent  to 
SYS  100  and  230  -  after  another  year  or  so,  they  should 
be  sent  to  SYS  225 

-  best  teacher  is  experience  for  both  TOMA  and 
acquisition  world  -  this  just  takes  time  (3) 

-  need  a  simplified  handbook  for  trainees  -  the  basic 
process  is  not  that  complicated  but  the  guidance  and 
regs  are  overwhelming 

-  need  a  good  training  plan  and  adding  797 's  to  TO 
manager's  623a  for  military  enlisted  to  follow 

-  all  TOMA  training  concepts  and  plans  have  been  "eye¬ 
wash  and  lip-service"  in  the  past 

Experience 

-  need  to  ensure  flight  line/maintenance  experience  of 
TOMA's  -  gives  them  a  better  feel  of  what  TO's  are 
needed  and  what  they  should  include 

-  eliminate  those  below  the  rank  of  TSGT  -  not  enough 
experience 

Career  Channel 

-  need  a  career  channel  for  TOMA's 

-  need  AFSC/skill  for  TOMA's 

*  -  military  TOMA's  should  be  protected  from  being  yanked 
out  of  their  career  fields  and  made  to  be  cops,  cooks, 
chapel  aides,  etc.  or  returned  to  their  old  career 
fields 

*  -  AFSC  should  be  established  to  ensure  TO  managers 
are  not  lost  due  to  PCS 

-  need  an  AFSC  for  TOMA's  at  the  grade  of  master 
sergeant  and  above 

Apprenticeship 

*  -  team  up  a  new  TOMA  with  an  experienced  one  (4) 

*  -  need  an  apprenticeship  program  to  cultivate  TOMA's 

*  -  more  interaction  and  supervision  by  an  experienced 

TOMA 

General 

-  need  "cradle  to  grave"  (source  selection  to  PMRT ) 
commitment  of  personnel  (TOMA's)  in  order  to  get 
experts 

-  centralized  TOMA  office  is  not  a  good  idea  -  each 
situation  is  different  -  especially  with  contractors 

-  ALC's  need  to  provide  slots  in  the  programs  to  allow 
their  individuals  to  learn  the  process  of  doing  TO 
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acquisition  and  support  equipment  -  need  to  rotate 

TOMA's  between  ALC  and  AF SC 

The  above  suggestions/comments  obviously  stress 
training  (with  emphasis  on  SYS  230  and  the  timeliness  of 
taking  the  classes)  and  an  apprenticeship  program  where  a 
more  experienced  TOMA  is  teamed  up  with  a  more  novice  TOMA. 
The  training  emphasis  seems  to  be  on  the  more  practical  side 
of  TOMA  duties,  l.e.,  "Exactly  what  am  I  supposed  to  be 
doing  day-to-day?"  There  was  also  a  mention  of  an  inability 
to  get  into  the  AFIT  classes  because  of  quotas.  The  TOMA's 
who  really  need  to  go  might  not  get  to  because  their 
organization  has  used  up  its  allotment  of  training  slots  for 
the  year. 

In  the  career  channel  grouping  above,  a  key  issue  was 
raised  -  retention.  It  was  suggested  that  military  TOMA's 
be  protected  from  being  pulled  into  other  career  fields  and 
that  an  Air  Force  Specialty  Code  (AF SC)  should  be 
established  to  ensure  TOMA's  are  not  "lost"  when  they  PCS. 
Another  benefit  of  the  TOMA  AFSC  would  be  the  record  of  TOMA 
experience  being  readily  available  in  the  personnel 
computer.  Presently,  if  the  Military  Personnel  Center  (MPC) 
wanted  to  know  who  had  TOMA  experience,  they  could  not  tell 
except  by  going  through  individual  performance  reports. 

Another  excellent  idea  brought  up  in  the  comments 
listed  above  was  to  have  some  kind  of  apprenticeship  program 
for  TOMA's.  This  could  be  within  a  SPO  if  it  is  a  large 
program  or  between  two  SPO's  at  relatively  the  same  place  in 
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the  acquisition  cycle  for  the  smaller  SPO's.  To  be  assigned 
to  a  more  experienced  TOMA  whom  one  could  call  upon  for 
advice  and  recommendations  would  be  a  tremendous  help  for 
the  less  experienced  TOMA's.  And  what  a  simple  and 
inexpensive  way  to  take  advantage  of  the  Air  Force  resources 
already  in  place. 

A  general  comment  listed  above  that  also  needs  to  be 
addressed  is  the  suggestion  that  the  TOMA's  be  rotated 
between  AFLC  and  AFSC  (Systems  Command).  This  would  allow 
the  AFLC  people  to  learn  early  the  process  of  TO 
acquisition.  It  would  probably  also  improve  AFLC/AFSC 
relations  since  working  side-by-side  can  often  lead  to  a 
greater  understanding  and  empathy  for  each  other.  This 
suggestion  would  be  much  more  expensive  because  of  the 
travel  and  per  diem  costs  associated  with  the  relocations. 

A  good  idea  but  only  if  there  are  sufficient  funds  to 
support  it  -  an  unlikely  case. 

Questionnaire  Question  16:  What  do  you  think  should  be 
included  in  the  criteria  for  selection  of  tech  order 
managers? 

Question  16  was  one  of  the  capstone  questions  of  the 
questionnaire.  It  is  asking  what  the  people  out  in  the 
trenches  think  should  be  the  basis  for  selection  for  the 
jobs  they  now  have.  There  was  quite  a  variety  of  answers, 
the  majority  of  which  are  depicted  in  Figure  15.  As  in 
question  6,  there  was  an  overlap  of  answers  and  the 
percentages  given  on  the  chart  are  based  on  the  total 
answers  covered  in  the  chart,  whereas  in  the  following 
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RANK  (14.1%)-^ 


PRIOR  TRG  (6.4%) 
COUP  KNOWL  (5.1%) 


NEGOTIATIONS  (7.7%)- 


-flELD  EXP  (34.6%) 


ACQ  EXP  (16.7%)- 


SYS230  (15.4%) 


Figure  15.  What  Should  Be  Included  in  the  Criteria  for  the 
Selection  of  Technical  Order  Managers 


elaboration  of  the  responses,  the  percentages  given  are 
bcsed  on  the  respondents.  For  example,  field  experience 
makes  up  35  percent  of  Figure  16  but  there  were  27 
respondents  citing  this  area  which  is  50  percent.  See  the 
explanation  in  question  6  for  further  clarification. 

Field  experience  was  by  far  the  most  common  requirement 
listed  with  50  percent  of  the  respondents  citing  this,  next 
was  acquisition  experience  with  24  percent  citing  its 
n^sessity,  followed  by  SYS  230  at  22  percent,  rank  at  19 
percent,  negotiations  experience  at  11  percent,  training 
prior  to  starting  TOMA  responsibilities  at  9  percent,  and 
knowledge  of  the  equipment  the  TOMA  is  responsible  for  with 
7  percent.  There  were  8  respondents  who  failed  to  answer 
this  question.  The  "other"  category  which  was  not  included 
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in  Figure  15  because  it  would  have  dwarfed  the  other  pie 
slices.  Included  the  following  (numbers  in  parentheses 
following  the  responses  indicate  the  frequency  of  response 
when  greater  than  one). 

-  contracting  process  (3) 

-  hard  working/motivated  (3) 

-  ability  to  travel  (3) 

-  business  practices/strategy  (2) 

-  configuration  background  (2) 

-  in-process-reviews  knowledge  (2) 

-  cost  evaluation  knowledge  (2) 

*  -  writing  ability  (2) 

*  --at  least  college-level  English  -  have  seen  TOMA's  who 
can  not  construct  good  sentences  or  spell  well  -  both 
of  which  should  be  second  nature  to  people  reviewing 
books  which  are  going  to  the  field 

-  speaking  ability  (2) 

-  validation/verification  experience  (2) 

-  be  able  to  work  well  with  and  communicate  with  people 
(2) 

-  common  sense  (2) 

-  SYS  100  (2) 

-  AFALC  001  (2) 

-  team  player  (2) 

-  data  item  description  experience 

-  statement  of  work  experience 

-  guidance  conference  experience 

-  tactfulness  in  dealing  with  others 

-  software  development  knowledge 

-  new  systems  knowledge 

-  computer  knowledge 

-  ability  to  think  on  one's  feet 

-  ability  to  handle  stress 

-  stability  to  stay  with  program 

-  ability  to  work  in  office  and  contractor  environment 

-  SYS  200/SAS  006 

-  one  week  at  the  main  operating  base 

-  experience  a  negotiation  with  a  fully  qualified  TOMA 

-  command  maintenance  understanding  -  TAC  does  things 
differently  from  SAC 

-  active  use  and  participation  in  TO  change  process 
( AFTO  22 's  etc) 

-  field  experience  helpful  but  not  necessary 

-  aggressive  and  willing  to  learn  regs,  contracts,  and 
timing 

-  understand  mil  specs  and  be  familiar  with  them 

-  SYS  225 

-  TO  file  maintenance  experience 

-  management  experience  training 
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-  administrative  experience  training 

-  minimum  of  an  associate's  degree 

-  clean  record 

-  volunteer 

-  hand  picked/interviewed 

-  fair  chance  at  the  job  -  advertize  in  Air  Force  unit 

bulletins  -  use  resumes 

-  should  not  get  job  simply  because  one  has  worked  with 

TO '  s 

-  extrovert  but  not  harsh  or  abrasive 

-  be  an  organizer 

-  team  player 

-  engineering  data  knowledge 

-  LSA  knowledge 

-  test  and  evaluation  knowledge 

An  expansion  is  in  order  on  the  "rank"  category  cited 
by  10  of  the  respondents.  One  respondent  thought  one  should 
not  become  a  TOMA  before  the  third  term  of  enlistment. 
Several  said  they  should  be  senior  NCO's  (non-commissioned 
officers)  or  civilian  GS-12's  or  above.  Several  said 
officers  should  not  be  TOMA's  because  they  are  usually  in 
the  job  for  too  short  of  an  amount  of  time  and  they  usually 
do  not  have  the  maintenance  background  required.  One 
commented  that  a  second  lieutenant  should  never  have  the  job 
because  they  do  not  usually  have  enough  experience.  Several 
mentioned  the  reluctance  of  contractors  to  deal  with  anyone 
with  a  rank  less  than  master  sergeant. 

Another  comment  that  needs  to  be  addressed  is  the 
writing  ability  of  the  TOMA.  It  was  suggested  that  TOMA's 
all  at  least  have  a  college-level  English  capability.  This 
requirement  is  certainly  a  reasonable  one  given  the 
extensive  reviewing  and  writing  tasking  placed  on  TOMA's. 
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Telephone  Interviews 


In  an  attempt  to  get  an  outside  perspective,  six 

defense  contractors  were  contacted  and  interviewed  by 

telephone  to  determine  their  selection  and  training  criteria 

for  technical  order  managers.  The  following  is  a  synopsis 

of  the  interviews,  grouped  by  question  -  not  contractor. 

Telephone  Interview  Question  1;  How  do  you  select  your 
TO  managers? 

Question  1  prompted  the  following  responses  from  the 
contractors.  (The  listed  responses  are  a  best  effort  at 
directly  quoting  the  respondents,  hence  the  sometimes 
awkward  syntax.) 

-  the  ideal  candidate  has  a  sound  technical  background, 
generally  in  electronics  ( West lnghouse ) ,  some  have 
computer  science  backgrounds 

-  engineering  writers  (as  opposed  to  technical 
writers),  a  lot  of  writers  have  engineering  degrees 

-  look  for  experience,  ability  to  relate  to  people,  NOT 
illustrators  -  we're  going  to  writers  with  an 
engineering  background,  want  people  who  will  like  the 
job 

-  look  for  people-skill  managers 

-  hired  from  technical  writers,  hired  strictly  on 
technical  skills  -  can  teach  them  how  to  write  but 
CAN'T  teach  the  technical  skills  -  longevity  is  a  big 
advantage 

-  hired  from  within,  hire  technically  oriented  people  - 
engineers,  technicians,  maintenance  people  with 
technical  backgrounds  -  turnover  in  Air  Force  is  a 
problem 

This  question  was  an  attempt  to  determine  how  the 
contractors  select  their  technical  order  managers  -  what 
they  considered  Important  when  looking  for  the  "right" 
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person.  There  was  quite  a  variety  of  answers  to  this 

question.  Most  of  the  contractors  are  looking  for  good 

technical  skills  because  they  say  they  can  teach  people  how 

to  write  but  they  cannot  teach  the  technical  skills. 

Several  commented  about  how  the  continuity  of  their  work 

force  was  a  big  advantage  over  the  more  mobile  Air  Force. 

They  know  what  their  people  have  been  doing  and  for  how  long 

whereas  some  of  the  Air  Force  personnel  being  considered  for 

jobs  as  TOMA's  have  been  on  such  varied  assignments  that 

they  are  almost  an  unknown  entity.  It  is  not  unusual  for 

contractor  personnel  to  stay  on  the  same  job  for  ten  years 

or  more  whereas  that  is  almost  unheard  of  in  the  Air  Force. 

Telephone  Interview  Question  2:  Are  there  selection 
criteria?  If  so,  what  are  they? 

Question  2  prompted  the  following  responses: 

-  Virtually  all  have  four  year  degrees 

-  Nothing  formal,  usually  want  a  degree  or  equivalent 

-  Not  necessary  -  all  people  have  at  least  eight  years 
of  technical  experience  and  two  years  of  direct 
supervisory  experience  before  they  are  eligible 

-  "grass  roots"  knowledge,  an  understanding  of 
automation,  experience  as  to  writers  or  illustrators 

-  strong  technical  skill  background  in  publications 
area,  extensive  experience  at  writing,  people  skills  to 
empower  people  and  enable  them  to  support  the  customer 

-  promote  the  best  technical  people  and  give  broadening 
assignments  to  them  in  contracting,  programmatic 
experience,  get  them  to  where  they  can  respond  for  a 
support  type  of  product  line 

This  question  was  trying  to  identify  any  structured  or 
formal  selection  criteria  the  contractors  might  use  in 
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selecting  their  technical  order  managers.  Surpr is ingly, 

there  were  no  formal  criteria  followed  by  any  of  the 

contractors.  There  were  some  "generally  followed"  rules 

such  as  four  year  degrees  or  the  equivalent  but  none  were 

strictly  followed  and  most  of  the  rules  of  thumb  were  very 

vague  such  as  "strong  technical  background."  The  general 

profile  followed  was  strong  technical  background  alone  and 

then  these  people  were  usually  given  some  career  broadening 

exposure  and  supervisory  experience  before  or  during  their 

first  assignment  as  a  technical  order  manager. 

Telephone  Interview  Question  3:  What  type  of 
background  for  a  TO  manager  have  you  found  to  be  most 
successful? 

Question  3  was  responded  to  as  follows: 

-  good  written  and  oral  communication  skills,  even 
liberal  arts  folks  can  succeed  with  some  technical 
background 

-  it  more  depends  on  the  individual,  a  lot  of  times  a 
technician  will  make  a  better  writer  than  an  engineer  - 
very  varied 

-  mechanical  comprehension/capability,  language  skills 

-  writing  and  verbal  and  meeting  ability 

-  hardworking,  good  people  skills,  good  work  ethic  - 
these  kind  of  people  migrate  toward  the  requirements 
section  and  do  technical  and  financial  negotiations  - 
mostly  OJT  to  fully  understand  the  work  required 

-  college  is  essential  coupled  with  the  pragmatics  of 
the  flight  line  and  broad  experience  -  have 
successfully  trained  an  undertaker,  electrical 
engineer,  mechanical  engineer,  technician  -  lots  of 
NCO's  with  lots  of  maintenance  experience 

This  question  attempted  to  determine  if  there  was  a 

common  background  or  denominator  for  the  "ideal"  contractor 

technical  order  manager.  The  backgrounds  of  the  most 
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successful  contractor  technical  order  managers  were  varied 
but  the  common  denominators  seemed  to  be  a  good  education 
(usually  defined  by  some  kind  -  not  necessarily  technical  - 
of  four  year  degree),  good  verbal  skills,  strong  work  ethic, 
and  strong  technical  background  or  understanding  of  the 
product . 

Telephone  Interview  Question  4;  What  type  of  training 
do  you  provide  for  your  TO  managers? 

Question  4  was  answered  as  follows: 

-  We  have  a  full  catalog  of  training  courses  with 
cross-actional  analysis  and  problem  solving  -  more  than 
250  courses 

-  No  formal  TO  training,  a  lot  of  management  training 

-  OJT  -  they're  all  graduates  of  a  two  year  technical 
school  or  have  had  prior  military  experience  or 
associated  technical  writing  experience 

-  at  the  writing  level  we  offer  writing  skills  classes, 
also  quality  analysis  and  materials  training  -  a  lot  of 
OJT  or  voluntary  courses  in  computer  literacy 

-  Used  to  have  week  long  formal  training  class  but  now 
are  in  a  lower  level  of  hiring  and  have  also  dropped 
the  training  course  -  use  strictly  OJT 

-  OJT  and  mentoring,  some  leadership  training  and  Total 
Quality  Management  training 

This  question  was  an  attempt  to  determine  the  types  of 
training  programs  provided  by  the  contractors.  The 
responses  varied  from  OJT  (on  the  job  training)  only  to 
extensive  classroom  opportunities.  The  question  falls  short 
of  really  determining  what  exactly  happens  and  concentrates 
more  on  what  is  "offered."  A  more  probing  interview  would 
be  necessary  to  find  out  how  often  the  technical  order 
managers  actually  do  go  to  classroom  training  and  how  easy 
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it  is  to  get  into  the  classes.  For  instance,  the  Air  Force 

could  respond  about  their  full  range  of  professional 

continuing  education  courses  offered  through  AFIT,  but  the 

Individual  Air  Force  TOMA's  have  some  def’nite  complaints 

about  their  inability  to  get  into  these  classes.  The  extent 

of  the  training  programs  within  the  contractors  polled 

seemed  to  vary  proportionately  with  the  size  of  their 

program.  If  the  people  interviewed  personally  managed  large 

dollar  defense  contracts,  their  technical  order  programs 

were  quite  extensive  and  if  they  handled  smaller  dollar 

contracts,  they  relied  mainly  on  OJT. 

Telephone  Interview  Question  5:  What  experience  or 
training  do  you  feel  is  most  essential  to  the  success  of  a 
TO  manager? 

Question  5  elicited  the  following  responses: 

-  Need  hands  on  experience,  day-to-day  maintenance 
definitely  -  need  that  along  with  high  technical  skills 

-  prior  military  experience,  more  than  a  four  year 
degree  -  combination  of  technical  skills  and  people  and 
business  schooling,  contracting  and  business  law  and 
accounting 

-  people  skills,  leadership,  interpersonal  skills,  time 
management 

-  quite  a  few  have  associate's  degrees  or  other  degrees 
in  business  -  lends  a  lot  to  their  capabilities 

-  to  know  that  LSA  (Logistics  Support  Analysis)  is  not 
a  cure-all 

This  question  attempted  to  determine  the  most  essential 
prerequisite  to  successful  technical  order  managers.  The 
responses,  as  can  be  seen  above,  were  quite  varied.  Hands- 
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on  experience,  people  skills  and  a  good  education  seemed  to 

be  the  common  denominators  however. 

Telephone  Interview  Question  6:  What  are  the  weak 
points  of  your  current  selection  and  training  methods? 

Strong  points? 

Question  6  prompted  the  following  responses: 

-  weak  -  financial  management  ability  -  need  more 
exposure  to  the  balance  sheet 

-  strong  -  supervisory  assessment  program  (This  is  a 
program  used  by  Westinghouse  Electric  Corporation  for 
assessing  and  training  supervisors  and  is  a  recommended 
area  for  further  research.) 

-  weak  -  don't  have  a  complete  TO  department  -  just 
small  groups  of  10  -  15  workers  -  not  as  big  a  pool  to 
choose  from 

-  strong  -  school  of  hard  knocks 

-  weak  -  exposure  to  business  aspects  comes  suddenly  to 
a  TO  manager  -  can't  really  prepare  anyone  for  it  -  no 
set  training 

-  strong  -  interviewing  process,  promotion  review  board 

-  if  the  person  dees  well  they'll  be  rewarded 

-  weak  -  vision  -  grappling  with  data  base  information 

-  the  wave  of  the  future  -  both  the  government  and 
contractors  are  struggling  to  avoid  bad  decisions 
regarding  computer  aided  logistics  systems 

-  strong  -  good  experience  -  employees  have  chosen  the 
field  as  a  career  and  have  been  in  the  field  for  a  long 
time 

-  weak  -  most  difficult  to  convince  people  to  change 
over  to  requirements  (the  office  that  primarily  deals 
with  customer  requirements)  -  no  more  money  -  same 
grade  -  just  more  challenging 

This  question  was  intended  as  a  catch-all  to  determine 
If  there  were  any  outstanding  features  to  be  emulated  or 
pitfalls  to  be  avoided  by  learning  from  the  contractors' 
experience.  Most  of  the  points  cited  are  familiar  to  the 
Air  Force  such  as  the  weakness  in  the  lack  of  vision  and  the 
strength  in  the  school  of  hard  knocks.  But  one  weakness 
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brought  up  by  several  contractors  and  not  given  much  weight 
in  the  Air  Force  either  is  the  lack  of  f inancial . background 
among  their  technical  order  managers.  This  was  an  area  of 
concern  for  the  contractors  because  they  operate  based  on 
the  bottom  line  -  they  have  to  make  a  profit  or  they  go  out 
of  business.  The  Air  Force  does  not  put  much  emphasis  on 
this  area  because  they  are  a  non-profit  organization.  This 
seems  to  be  a  disconnect  because  if  the  Air  Force  is 
negotiating  with  people  whose  objective  is  the  bottom  line, 
it  should  follow  that  they  would  put  much  emphasis  on 
understanding  the  contractors'  viewpoint  in  order  to  better 
negotiate  with  them.  Some  strengths  cited  that  the  Air 
Force  does  not  share  with  the  contractors  are  the 
interviewing  process  and  the  depth  of  experience. 

One  of  the  strengths  mentioned  by  Westinghouse  Electric 
Corporation  is  the  use  of  a  supervisory  assessment  program 
for  evaluating  the  management  skills  of  potential 
supervisors  in  order  to  make  recommendations  and  prognoses 
on  the  supervisors'  performances.  According  to  Mr.  Mike 
Pfeifer  of  Westinghouse  in  Hanover,  Maryland,  this  type  of 
program  has  been  widely  used  in  industry  for  the  last  30  to 
40  years.  It  was  devised  in  World  War  II  by  the  military  to 
select  the  best  officers  for  promotion  and  command.  It 
involves  role  playing  in  individual  and  group  exercises  that 
are  indicative  of  the  future  problems  the  potential 


73 


supervisors  might  face.  The  company  that  Westlnghouse 
contracts  with  to  sponsor  the  program  is  Development 
Dimensions  International  in  Pittsburg,  Pennsylvania.  The 
person  who  presently  heads  the  department  that  Westlnghouse 
works  with  is  Mr.  Bill  Byham  and  he  can  be  reached  at  (412) 
257-0600. 

Chapter  Summary 

In  the  preceding  chapter,  the  responses  to  the  ASD 
technical  order  managers'  questionnaires  and  the  responses 
to  the  contractors'  telephone  interviews  have  been  reported 
and  analyzed  with  respect  to  the  research  objectives.  These 
findings  and  analyses  will  be  used  to  draw  conclusions  and 
make  recommendations. 
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V.  Conclusions  and  Recommendat ions 

This  chapter  addresses  the  significance  of  the  results 
of  the  survey  and  the  practical  implications  of  those 
results.  Recommendations  for  improved  technical  order 
manager  selection  and  training  methods  as  well  as  follow-on 
or  revised  studies  are  then  made. 

Conclusions 

The  results  of  the  survey  are  significant  in  that  they 
confirm  most  of  the  premises  on  which  the  thesis  is  based 
and  they  also  suggest  some  critical  areas  to  be  looked  at  in 
terms  of  improvement  of  the  present  technical  order  manager 
selection  and  training  process  within  Aeronautical  Systems 
Division.  Based  on  the  research  conducted,  it  was 
determined  that  ASD  does  not  have  a  selection  criteria  for 
selcecting  TOMA's  nor  does  it  have  a  formal  training  program 
for  the  TOMA's.  However,  ASD/ALXS  is  actively  developing  a 
training  program. 

The  following  conclusions  and  recommendations  are  based 
on  the  responses  to  the  questionnaire  completed  by  64.3 
percent  of  all  ASD  technical  order  managers  as  well  as  the 
telephone  interviews  conducted  with  six  major  defense 
contractors.  The  general  topic  order  of  conclusions  and 
recommendations  follows  the  same  order  as  the  question 
topics  addressed  in  the  questionnaire. 
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Grade  of  TOMA's.  The  survey  results  indicated  that  the 


majority  of.  the  ASD  technical  order  managers  are  enlisted 
(63  percent).  This  is  significant  in  terms  of  training 
allotments  by  grade.  Perhaps  the  applicable  training 
allotments  (exactly  what  is  applicable  will  be  addressed  in 
a  later  paragraph)  should  be  reviewed  to  ensure  that  they 
coincide  with  the  grade  distribution  of  technical  order 
managers.  The  large  percentage  of  military  technical  order 
managers  (72.3  percent)  should  perhaps  also  be  analyzed  in 
terms  of  continuity,  i.e.,  does  being  military  degrade  the 
continuity  of  the  TOMA  or  are  the  civilians  equally  as 
mobile  among  the  different  SPO's?  Continuity  was  a  big 
issue  according  to  the  contractors  and  one  of  their  best 
strong  points. 

Education  Level  of  TOMA’s.  Education  level  also  seems 
to  be  important  to  the  contractors.  Many  insist  on  a  four 
year  college  degree  of  some  kind  or  an  equivalent.  More 
basic  than  the  college  degree  though,  are  the  essential 
writing  and  communication  skills  required  of  all  TOMA's. 
Several  comments  were  made  by  the  TOMA's  that  basic  writing 
skills  need  to  be  improved  -  at  least  to  a  college  English 
level  due  to  the  large  amount  of  writing  and  effective 
communication  skills  required  of  all  TOMA's.  Perhaps  a 
writing  effectiveness  test  should  be  administered  through 
ASD/ALXS  once  one  is  chosen  as  a  TOMA,  and  if  the 
performance  is  substandard,  a  rudimentary  English  class 
should  be  required.  AFIT  offers  these  kinds  of  tests  and 
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classes  for  their  new  graduate  students  as  do  many  local 
area  colleges  and  universities. 

Prior  Work  Experience  As  TOMA's.  The  majority  (63 
percent)  of  the  TOMA's  had  more  than  two  years  of  TOMA- 
related  experience.  This  experience  lends  credibility  to 
the  survey  results  because  most  of  the  responding  TOMA's 
have  been  TOMA's  long  enough  to  make  knowledgeable 
recommendations.  Because  questionnaire  question  3  did  not 
differentiate  between  total  experience  as  a  TOMA  and 
experience  as  a  TOMA  only  within  their  present  program,  the 
issue  of  continuity  (which  is  brought  up  by  several 
respondents)  cannot  be  addressed  in  light  of  the 
questionnaire  data,  i.e.,  the  percentages  of  TOMA's  with 
more  than  two  years  experience  in  their  present  programs 
cannot  be  determined  from  the  questionnaire  data.  The 
continuity  issue  can  be  addressed,  however,  in  general 
terms,  based  on  the  comments  of  several  respondents. 

Dollar  Value  of  Technical  Orders  Managed.  Most  (66 
percent)  of  the  TOMA's  manage  more  than  $1  million  worth  of 
technical  orders.  On  the  basis  of  financial  responsibility 
alone,  it  is  worthwhile  to  ensure  these  technical  order 
managers  are  as  carefully  selected,  experienced  and  trained 
as  possible. 

Number  of  Technical  Orders  Managed.  The  number  of 
technical  orders  managed  is  not  a  good  indicator  of  the 
level  of  responsibility  assigned  to  the  TOMA.  This  is 
because  the  number  of  technical  orders  is  not  necessarily 
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directly  related  to  the  dollar  value  or  the  amount  of  effort 
required  to  properly  manage  the  technical  orders. 

Prior  Technical  Order  Experience.  Most  of  the 
responding  TOMA's  had  either  flight  line  (67  percent)  and/or 
in-shop  (59  percent)  experience  working  with  technical 
orders . 

Related  Training  Courses  Completed.  Most  of  the 
respondents  had  taken  SYS  100  (76  percent)  and/or  SYS  230 
(83  percent).  These  are  encouragingly  high  percentages 
given  the  fundamental  Importance  of  these  courses  in 
acquisition  and  technical  order  management.  Although 
encouraging,  there  is  still  room  for  improvement. 

Technical  Order-Related  Experience  Qualifying  the 
TOMA's  for  Their  Job3.  Most  (72  percent)  of  the  TOMA'3 
thought  the  technical  order-related  experience  which 
qualified  them  for  their  jobs  was  their  hands-on  experience 
using  technical  orders. 

Prior  Experience  That  Would  Have  Better  Prepared  the 
TOMA's .  Most  of  the  answers  (75  percent)  given  by  the 
respondents  indicated  that  the  respondents  felt  they  would 
have  been  better  prepared  if  they  had  had  more  experience  in 
the  typical  SPO  activities  such  as  in-process  reviews, 
guidance  conferences,  negotiations,  statements  of  work  and 
data  item  description  reviews.  Those  who  did  not  already 
have  hands-on  experience  using  technical  orders  felt  that 
experience  would  have  better  prepared  them  for  their  jobs. 
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Training  Desired  To  Improve  Present  Job  Performance. 


If  the  TOMA's  have  not  had  SYS  230  (Air  Force  Technical 
Order  Acquisition  and  Management)  or  SYS  100  (Introduction 
to  Acquisition  Management),  they  want  to  take  the  courses. 
Otherwise,  SYS  225  (Acquisition  Logistics),  SYS  028 
(Introduction  to  Configuration  Management),  and  SYS  229 
(Test  and  Evaluation)  are  the  courses  the  TOMA's  most  desire 
to  Improve  their  present  job  performances. 

Shortcomings  Witnessed  In  Other  TOMA's.  The 
shortcomings  cited  were  primarily  in  the  SPO-related 
activities  field  although  the  lack  of  knowledge  of  technical 
orders  as  a  shortcoming  witnessed  in  other  TOMA's  was  not 
insignificant.  Again,  this  seems  to  point  to  shortcomings 
in  the  SPO  background  and  understanding  of  toma's 
responsibilities. 

Strong  Points  Witnessed  In  Other  TOMA'3.  Knowledge  of 
technical  orders  is  considered  to  be  a  TOMA  strong  point. 
Tactfulness  in  dealing  with  others  is  another  strong  point 
according  to  the  ASD  TOMA's.  The  later  strong  point  needs 
to  receive  more  emphasis  based  on  its  strong  showing  among 
the  ASD  TOMA's  who  consider  it  essential  to  good  TOMA 
performance . 

Duties  of  the  TOMA.  The  ASD  TOMA's  duties  are  many  and 
varied,  primarily  consisting  of  ln-process-reviews, 
statements  of  work,  validation/verification,  negotiations, 
data  item  descriptions,  cost  evaluations,  and  pre/post 
publication  reviews. 
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Whv  the  TOMA's  Were  Selected  For  Their  Jobs.  Most  of 
the  ASD  TOMA's  feel  they  were  selected  for  their  present 
jobs  on  the  basis  of  their  experience  and/or  maintenance 
background.  Quite  a  few  felt  they  were  chosen  on  the  basis 
of  their  being  available  or  a  volunteer.  As  the  TOMA 
selection  process  improves,  this  question  should  reflect 
more  answers  along  the  lines  of  experience  and  training  and 
less  along  the  lines  of  luck. 

How  To  Improve  the  Training  of  TOMA's.  The  TOMA's 
indicated  a  strong  desire  for  more  practical  training,  i.e., 
how  to  actually  do  their  jobs  done  as  opposed  to  the  history 
of  technical  orders.  It  was  suggested  that  practical 
experiences  such  as  conducting  in-process  reviews  and 
guidance  conferences  be  actually  played  out  in  the  classes. 
There  also  is  a  strong  desire  for  Informal  seminars  (that  do 
not  require  a  training  quota  reserved  far  in  advance)  to 
address  current  Issues  of  concern  to  TOMA's. 

The  Selection  Criteria  for  TOMA's.  The  ASD  TOMA's 
believe  the  selection  criteria  should  still  be  largely  based 
on  hands-on  experience  working  with  technical  orders.  They 
also  feel  that  the  acquisition  background  is  very  important 
for  TOMA's  and  they  realize  that  to  find  this  combination  of 
experiences  is  difficult  at  best.  The  best  course  of  action 
then  is  to  select  the  potential  TOMA's  based  on  their  field 
experience  and  grade  and  then  have  a  structured,  realistic 
training  program  and  apprenticeship  approach  to  best  educate 
and  guide  the  TOMA's  through  their  early  years. 
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Recommendations 


Selection  Criteria. 

Grade .  TOMA's  should  be  of  a  sufficient  grade  to 
give  them  credibility  when  they  are  dealing  with  the 
contractors.  Recommended  minimum  grades  are  Master  Sergeant 
if  enlisted,  a  GS-12  if  civilian,  and  a  Captain  if  and 
officer  unless  the  officer  has  prior  enlisted  experience  in 
which  case  they  could  be  a  lieutenant. 

Education  Level.  A  bachelor's  degree  should  be 
strongly  encouraged,  there  by  ensuring  at  least  a  college- 
level  competency  in  English.  As  a  minimum,  a  competency 
test  should  be  given  to  all  TOMA's  whether  or  not  they  have 
a  college  degree,  measuring  their  ability  to  write  well- 
constructed  sentences  and  evaluate  the  same.  If  the  TOMA 
demonstrates  a  need  for  improved  writing  skills,  alternative 
sources  to  improve  their  skills  should  be  offered  such  as 
courses  already  offered  through  AFIT,  or  some  of  the 
surrounding  colleges  and  universities  (Wright  State 
University,  University  of  Dayton,  Sinclair  College,  etc.). 

Prior  Experience.  The  most  essential  requirement 
of  a  successful  technical  order  manager  is  a  basic  and 
thorough  understanding  of  technical  orders  and  how  they  are 
used.  For  this  reason,  it  is  recommended  that  the  primary 
basis  for  selection  of  technical  order  managers  continue  to 
be  hands-on  experience  using  them.  The  SPO  functions  and 
acquisition  techniques  can  be  learned  with  adequate  methods 
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as  recommended  herein,  but  the  basic  technical  background 
and  understanding  must  come  from  experience. 


Career  Channel.  A  career  channel  is  needed  for 
TOMA's.  This  would  involve  establishing  a  separate  Air 
Force  Specialty  Code  for  military  and  a  separate  job  series 
identifier  for  civilians  (this  is  also  an  area  recommended 
for  further  research).  This  would  allow  the  respective 
personnel  departments  to  easily  query  their  systems  for 
lists  of  persons  with  TOMA  experience.  The  TOMA  career 
field  would  then  need  to  be  given  some  kind  of  priority  to 
make  it  immune  from  the  often  indiscriminate  requirements 
for  undermanned  or  hard  to  fill  positions  such  as  security 
police,  chaplain  aides  or  cooks. 

Continuity.  To  improve  the  continuity  of  TOMA's, 
a  policy  should  be  established  or  perhaps  even  a  contract 
signed  between  the  personnel  establishment  and  the  new  TOMA, 
obligating  the  TOMA  for  at  least  four  years  to  that  program 
office.  (The  life  of  the  program  would  be  preferable  but 
hard  to  insist  upon.) 

Training . 

SYS  230  and  SYS  100.  These  two  classes  are  the 
bare  essentials  for  any  new  TOMA.  They  should  be  attended 
by  all  TOMA's  after  their  first  six  months  but  prior  to  the 
end  of  their  first  year  as  a  TOMA.  They  should  not  be 
scheduled  immediately  upon  becoming  a  TOMA  because  the 
TOMA's  need  to  have  enough  time  on  the  job  for  the  classroom 
teachings  to  be  meaningful.  TOMA's  should  be  given  priority 


for  SYS  230  over  all  other  attendees  regardless  of  quota 
allotments . 

Interpersonal  Relations.  It  is  recommended  that 
renewed  emphasis  be  placed  on  this  often  overlooked  area. 
Classes  in  the  base  Social  Actions  office  could  be  looked  at 
to  see  if  they  address  the  importance  of  tactfulness  in 
dealing  with  people.  Or,  as  suggested  above  regarding  the 
college-level  English  competency, . area  colleges  and 
universities  offer  many  of  these  interpersonal  relationship 
type  classes  and  attendance  at  these  classes  should  be 
encouraged . 

Training  Quotas.  The  quotas  allotted  to  the 
different  organizations  for  the  attendance  of  training 
classes  should  be  periodically  reviewed  to  ensure  they 
coincide  with  the  numbers  and  grades  of  the  TOMA's  who  have 
not  yet  attended  SYS  230. 

Apprenticeship  Program.  Every  new  TOMA  should  be 
paired  up  with  another  more  experienced  TOMA  who  has  had 
experience  at  the  same  stage  of  the  acquisition  life  cycle 
as  the  new  TOMA  is  now  in.  The  senior  TOMA  could  be  in  the 
same  program  office  for  those  larger  programs  or  just  in  a 
similar  program  for  those  in  smaller  SPO's.  This 
apprenticeship  relationship  should  be  a  great  benefit  to  the 
new  TOMA's  especially  during  the  first  six  months  or  until 
they  are  able  to  get  to  the  SYS  230  and  SYS  100  classes. 

This  is  a  relatively  inexpensive  program  with  great 
potential  benefit. 
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Providing  Hands-On  Experience.  For  those  TOMA's 
who  have  not  had  a  sufficient  or  perhaps  even  any  actual 
hands-on  experience  working  with  technical  orders,  they 
could  still  get  this  experience  without  having  to  move  to 
another  base.  An  agreement  could  possibly  be  worked  out 
with  the  4950th  Test  Wing  in  Area  C  of  Wr ight-Patterson  Air 
Force  Base  to  cooperate  in  providing  broadening  experience 
tours  (much  like  those  tours  sponsored  by  Systems  Command  - 
under  the  same  name  -  for  acquisition  officers  and 
engineers)  for  those  TOMA ' s  who  need  the  hands-on 
experience . 

Further  Recommendations  For  Research 

In  order  to  get  another  perspective  on  what  is 
important  as  far  as  the  training  and  selection  criteria  of 
technical  order  managers,  it  is  recommended  that  their 
supervisors  also  be  polled.  They  might  have  differing 
opinions  on  what  is  and  is  not  important. 

An  excellent  and  timely  topic  in  which  to  do  further 

research  would  be  the  specialty  codes  identifying  technical 
order  managers  within  the  personnel  system.  The  Air  Force 

Specialty  Code  (AFSC)  establishment  procedures  could  be 
studied  as  well  as  the  procedures  for  establishing  a 
civilian  job  series  identifier.  Details  could  be  looked  at 
and  then  recommendations  made  as  to  what  kind  and  how  much 
experience  should  warrant  obtaining  these  specialty  codes. 
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If  another  questionnaire  similar  to  tne  one  conducted 
herein  is  administered,  it  is  recommended  that 
"negotiations”  be  specifically  defined.  A3  the  term  was 
used  in  the  ASD  TOMA  questionnaire,  it  could  be  interpreted 
in  several  different  ways. 

Again,  if  another  questionnaire  similar  to  the  one 
conducted  herein  is  administered,  the  questions  should  be 
made  as  structured  as  possible  to  aid  in  ease  of  data 
compilation  and  analysis.  The  frequently  offered  responses 
to  the  unstructured  questions  could  be  offered  as  options  in 
the  newer  questionnaire. 

And  finally,  the  supervisory  assessment  program  that  is 
used  by  Westlnghouse  Electric  Corporation  for  assessing  and 
training  their  supervisors  should  be  further  investigated. 
This  is  a  sophisticated  and  successful  program  used  by 
industry  to  evaluate,  make  recommendations,  and  provide 
prognoses  for  potential  supervisors.  The  Air  Force  could 
greatly  benefit  from  the  usage  of  this  system. 

Chapter  Summary 

This  study  reviews  the  general  issue  of  inadequate 
training  and  selection  criteria  for  technical  order  managers 
in  Aeronautical  Systems  Division.  Based  on  questionnaire 
responses  from  the  Aeronautical  Systems  Division  technical 
order  managers  as  well  as  telephone  interview  responses  from 
six  defense  contractor  technical  order  managers,  conclusions 
were  drawn  about  the  present  status  of  the  training  and 
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selection  criteria.  Recommendations  were  then  made  on  how 
to  Improve  the  training  and  selection  criteria  for  the  ASD 
technical  order  managers. 


86 


15  Mar  90 


LSG 

Questionnaire  for  ASD  Technical  Order  Managers 


1.  I  am  a  graduate  student  at  the  Air  Force  Institute  of 
Technology  and  have  chosen  to  focus  my  thesis  research  on 
how  to  improve  the  selection  criteria  and  training  for 
technical  order  managers  in  Aeronautical  Systems  Division. 

I  am  working  with  ASD/ALXS  in  an  effort  to  improve  the 
present  shortfall  in  this  area. 

2.  I'm  trying  to  find  out  from  you,  the  technical  order 
managers,  how  you  can  be  better  prepared  and  trained  for  the 
important  jobs  that  you  do.  You  know  better  than  anybody 
else  what  kind  of  training  and  background  are  essential  in 
the  crucial  world  of  tech  order  management. 

3.  Please  take  the  time  to  fill  out  the  attached 
questionnaire  and  return  it  to  me  in  the  attached  return 
envelope  within  five  working  days  of  receipt.  We  will  be 
using  your  input  to  determine  the  best  combination  of 
selection  criteria  and  training  methods  for  all  ASD 
technical  order  managers.  Your  participation  is  completely 
voluntary,  but  your  Input  is  important! 

4.  Your  individual  responses  will  be  combined  with  others 
and  will  not  be  attributed  to  you  personally.  If  you  have 
any  questions  or  concerns,  please  call  me  at  extension 
54437. 

5.  Thank  you  for  your  assistance  in  improving  the  vital 
area  of  technical  order  management. 


NANCY  M.  DEMING,  Captain,  USAF  2  Atchs 

School  of  Systems  and  Logistics  1.  Questionnaire 

Air  Force  institute  of  Technology  2.  Return  Envelope 
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ASP  Tech  Order  Manager  Questions 


1)  What  is  your  grade? 


1)  0-1/0-2 

2)  0-3 

3)  0-4 

4)  0-5  or  above 

5)  GS-7/8 

6)  GS-9/10 

7)  GS-11 

8)  GS-12 


9) 

10) 

11) 

12) 

13) 

14) 

15) 


GS/GM-13  or  above 

E-5  or  below 

E-6 

E-7 

E-8 


E-9 

Other  (Please  specify  at  end  of 
survey. ) 


2)  What  is  .the  highest  level  of  education  you  have  achieved? 

1)  Did  not  complete  high  school 

2)  High  school  diploma  or  equivalent 

3)  Some  college  (Completed  at  least  one  college  course  with 
a  grade  of  C  or  above — excluding  Professional  Military 
Education . ) 

4)  Associate  degree 

5)  Bachelor  degree 

6)  Some  graduate  work  (You  already  have  a  batchelor  degree 
and  you  have  completed  at  least  one  graduate  level  course 
with  a  grade  of  B  or  above.) 

7)  Master  degree 

8)  Doctoral  degree 

9)  Other  (Please  specify  at  end  of  survey.) 


3)  How  long  have  you  been  working  as  a  technical  order 
manager? 

1)  Less  than  3  months 

2)  3-9  months 

3)  More  than  9  months  but  not  more  than  a  year 

4)  Between  1  and  2  years 

5)  2  years  or  more 

4)  what  is  the  total  dollar  value  of  the  tech  orders  you 
manage? 

1)  Less  than  $25,000 

2)  $25,000  -  $99,999 

3)  $100,000  -  $249,999 

4)  $250,000  -  $999,999 

5)  $1,000,000  -  $4,999,999 

6)  $5,000,000  -  $24,999,999 

7)  $25,000,000  or  more 

8)  Don't  know 
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5)  Approximately  how  many  technical  orders  do  you  manage? 


1)  1-100  3)  501-1000  5)  2001-3000  7)  More  than  5000 

2)  101-500  4)  1001-2000  6)  3001-5000  8)  Don't  know 

6)  In  what  capacity  have  you  worked  with  tech  orders  prior 
to  your  present  job  (flight  line,  maintenance,  in-shop,  in 
the  System  Program  Office,  etc.)? 


7)  What  kind  of  technical  order  acquisition  manager-related 
training  have  you  had? 

1)  SYS  230  AF  Technical  Order  Acquisition  and  Management 

2)  SYS  100  Intro  to  Acquisition  Management 

3)  SYS  225  Acquisition  Logistics 

4)  SYS  028  Intro  to  Configuration  Management 

5)  SYS  229  Test  and  Evaluation 

6)  AFALC  001  Deputy  Program  Manager  for  Logistics  ( DPML ) 
Course 


8)  What  type  of  experience  did  you  have  to  qualify  you  for 
this  job? 

1)  Preparation  of  Statements  of  Work 

2)  Negotiations 

3)  Guidance  Conferences 

4)  In-Process  Reviews 

5)  Hands-on  experience  using  technical  orders 

6)  Data-ltem-descr lpt ion  Reviews 

7)  Other  (Please  elaborate  at  end  of  questionnaire.) 


9)  What  kind  of  experience,  prior  to  your  coming  on  the  job, 
would  have  better  prepared  you  for  the  job  you  have  now? 

1)  Preparation  of  Statements  of  Work 

2)  Negotiations 

3)  Guidance  Conferences 

4)  In-Process  Reviews 

5)  Hands-on  experience  using  technical  orders 

6)  Data-item-descr iption  Reviews 

7)  Other  (Please  elaborate  at  end  of  questionnaire.) 
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10)  What  kind  of  training  would  you  still  like  to  have  in 
order  to  improve  your  present  level  of  job  performance? 

1)  SYS  100  Introduction  to  Acquisition  Management 

2)  SYS  028  Introduction  to  Configuration  Management 

3)  SYS  225  Acquisition  Logistics 

4)  SYS  229  Test  and  Evaluation 

5)  AFALC  001  Deputy  Program  Manager  for  Logistics  ( DPML ) 
Course 

6)  SYS  230  AF  Technical  Order  acquisition  and  Management 

7)  Other  (Please  specify  at  the  end  of  the  questionnaire.) 

11)  What  shortcomings  have  you  witnessed  in  other  technical 
order  managers? 

1)  Lack  of  acquisition  background 

2)  Lack  of  conf Iguratlon  background 

3)  Lack  of  negotiation  experience 

A)  Lack  of  Data  Item  Description  experience 

5)  Lack  of  knowledge  of  technical  orders 

6)  Lack  of  In-Process  Review  experience 

7)  Lack  of  Statement  of  Work  experience 

8)  Lack  of  Guidance  Conference  experience 

9)  Lack  of  tactfulness  in  dealing  with  people 

10)  Lact  of  cost  evaluation  experience 

11)  Other  (Please  elaborate  at  the  end  of  questionnaire.) 


12)  What  strong  points  have  you  witnessed  in  other  tech 
order  managers? 

1)  Acquisition  background 

2)  Configuration  background 

3)  Knowledge  of  negotiation  techniques 

4)  Data  Item  Description  experience 

5)  Knowledge  of  technical  orders 

6)  In-Process  Review  experience 

7)  Statement  of  Work  experience 

8)  Guidance  Conference  experience 

9)  Tactfulness  In  dealing  with  people 

10)  Cost  evaluation  experience 

11)  Other  (Please  elaborate  at  the  end  of  the 
questionnaire. ) 
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13)  Please  try  to  list  and  explain,  if  necessary,  all  of 
your  duties  associated  with  technical  order  management  (such 
as  negotiations,  in-process-reviews,  data  item  description 
review,  etc . ) 


14)  Why  do  you  think  you  were  you  selected  for  this  job? 
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15)  Is  there  anything  that  you  would  like  to  add  that  might 
help  improve  the  training  of  tech  order  managers?  If  so, 
please  elaborate. 


16)  What  do  you  think  should  be  included  in  the  criteria  for 
selection  of  tech  order  managers?  (Examples:  field 
experience.  Systems  230,  negotiation  experience,  etc.) 
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END  OF  QUESTIONNAIRE 


In  the  space  provided  below,  please  elaborate  on  any 
questions  In  which  you  selected  "Other”  for  a  response. 
Please  write  the  question  number  and  then  state  your 
elaboration . 
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Appendix  B:  Reminder  Letter  to  ASP  TOMA’s 


1  May  90 


This  Is  just  a  reminder  for  those  of  you  who  have  received 
my  questionnaire  on  tech  order  management  issues  but  have 
not  yet  taken  time  to  fill  It  out.  It's  not  too  late! 

Some  of  you  filled  out  a  preliminary  quest ionnaire  for  me. 

If  you  did,  I  still  need  you  to  fill  out  this  one.  The 
preliminary  one  was  just  to  work  out  the  bugs  and  make  sure 
my  questions  were  asking  what  I  wanted  them  to  ask. 

If  you  have  already  filled  out  the  questionnaire  and 
returned  it  to  me,  please  ignore  this  reminder  and  place 
this  paper  in  your  nearest  recycling  bin. 

If  you  haven't  received  the  questionnaire  or  have  misplaced 
it,  please  give  me  a  call  at  extension  58989  or  878-6706  and 
I’ll  be  glad  to  forward  a  new  one  to  you. 

To  date,  about  50  percent  of  the  questionnaires  have  been 
returned  to  me.  Please  take  the  time  and  opportunity  to 
voice  your  opinion  about  the  tech  order  management 
situation . 

Thanks  again  for  your  help. 


Nancy  M.  Deming,  Captain,  USAF 
School  of  Systems  and  Logistics 
Air  Force  Institute  of  Technology 
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Appendix  C:  Summary  of  Questionnaire  Responses 


ASP  Tech  Order  Manager  Questions  I  The  numbers  in 
parentheses  to  the  right  of  the  answers  are  the  numbers  of 
respondents  (54  total)  that  selected  the  answer.) 


1) 

What  is 

your 

grade? 

1) 

0-1/0-2 

(2) 

9) 

GS/GM-13  or  above 

(3) 

2) 

0-3 

(2) 

10) 

E-5  or  below 

(3) 

3) 

0-4 

(1) 

11) 

E-6 

(12) 

4) 

0-5  or 

above 

(0)  12) 

E-7 

(14) 

5) 

GS-7/8 

(0) 

13) 

E-8 

(3) 

6) 

GS-9/10 

(0) 

14) 

E-9 

(2) 

7) 

GS-11 

(2) 

15) 

Other  (Please  specify  at 

8) 

GS-12 

(10) 

survey) 

(0) 

2)  What  is  the  highest  level  of  education  you  have  achieved? 

1)  Did  not  complete  high  school  (1) 

2)  High  school  diploma  or  equivalent  (7) 

3)  Some  college  (Completed  at  least  one  college  course  with 
a  grade  of  C  or  above--excluding  Professional  Military 
Education.)  (20) 

4)  Associate  degree  (15) 

5)  Bachelor  degree  (5) 

6)  Some  graduate  work  (You  already  have  a  bachelor's  degree 
and  you  have  completed  at  least  one  graduate  level  course 
with  a  grade  of  B  or  above.)  (4) 

7)  Master  degree  (2) 

8)  Doctoral  degree  (0) 

9)  Other  (Please  specify  at  end  of  survey.)  (0) 


3)  How  long  have  you  been  working  as  a  technical  order 
manager? 

1)  Less  than  3  months  (1) 

2)  3-9  months  (4) 

3)  More  than  9  months  but  not  more  than  a  year  (4) 

4)  Between  1  and  2  years  (11) 

5)  2  years  or  more  (34) 
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4)  What  is  the  total  dollar  value  of  the  tech  orders  you 
manage? 


1)  Less  than  $25,000  (4) 

2)  $25,000  -  $99,999  (0) 

3)  $100,000  -  $249,999  (0) 

4)  $250,000  -  $999,999  (4) 

5)  $1,000,000  -  $4,999,999  (13) 

6)  $5,000,000  -  $24,999,999  (6) 

7)  $25,000,000  or  more  (22) 

8)  Don't  know  (7) 


5)  Approximately  how  many  technical  orders  do  you  manage? 

1)  1-100  (20)  3)  501-1000(5)  5)  2001-3000(5)  7)  More  than 

5000(0) 

2)  101-500  (11)  4)  1001-2000(5)  6)  3001-5000(2)  8)  Don't 

know( 6 ) 


6)  In  what  capacity  have  you  worked  with  tech  orders  prior 
to  your  present  job  (flight  line,  maintenance,  in--shop,  in 
the  System  Program  Office,  etc.)? 

Flight  Line  (36)  Misc  (15) 

In-Shop  (32)  Training  (3) 

SPO  (6)  None  (5) 

Val/Ver  (4) 

Staff  (3) 


7)  What  kind  of  technical  order  acquisition  manager-related 
training  have  you  had? 

1)  SYS  230  AF  Technical  Order  Acquisition  and  Management 
(45) 

2)  SYS  100  Intro  to  Acquisition  Management  (41) 

3)  SYS  225  Acquisition  Logistics  (7) 

4)  SYS  028  Intro  to  Configuration  Management  (7) 

5)  SYS  229  Test  and  Evaluation  (0) 

6)  AFALC  001  Deputy  Program  Manager  for  Logistics  ( DPML ) 
Course  (19) 

None/No  Response  (Not  an  answer/choice )  (3) 
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8)  What  type  of  experience  did  you  have  to  qualify  you  for 
this  job? 

1)  Preparation  of  Statements  of  Work  (22) 

2)  Negotiations  (6) 

3)  Guidance  Conferences  (15) 

4)  In-Process  Reviews  (18) 

5)  Hands-on  experience  using  technical  orders  (39) 

6)  Data-item-descrlptlon  Reviews  (7) 

7)  Other  (Please  elaborate  at  end  of  questionnaire.)  (15) 
None/No  Response  (Not  an  answer/choice)  (3) 


9)  What  kind  of  experience,  prior  to  your  coming  on  the  job, 
would  have  better  prepared  you  for  the  job  you  have  now? 

1)  Preparation  of  Statements  of  Work  (22) 

2)  Negotiations  (24) 

3)  Guidance  Conferences  (23) 

4)  In-Process  Reviews  (21) 

5)  Hands-on  experience  using  technical  orders  (14) 

6)  Data-item-description  Reviews  (18) 

7)  other  (Please  elaborate  at  end  of  questionnaire.)  (18) 


1C)  What  kind  of  training  would  you  still  like  to  have  in 
order  to  improve  your  present  level  of  job  performance? 

1)  SYS  100  Introduction  to  Acquisition  Management  (7) 

2)  SYS  028  Introduction  to  Conf iguratior  Management  (24) 

3)  SYS  225  Acquisition  Logistics  (35) 

4)  SYS  229  Test  and  Evaluation  (19) 

5)  AFALC  001  Deputy  Program  Manager  for  Logistics  ( DPML ) 
Course  (14) 

6)  SYS  230  AF  Technical  Order  Acquisition  and  Management 

(11) 

7)  Other  (Please  specify  at  the  end  of  the  questionnaire.) 

(14) 

No  Response  (4) 
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11)  What  shortcomings  have  you  witnessed  In  other  technical 
order  managers? 

1)  Lack  of  acquisition  background  (39) 

2)  Lack  of  configuration  background  (21) 

3)  Lack  of  negotiation  experience  (33) 

4)  Lack  of  Data  Item  Description  experience  (28) 

5)  Lack  of  knowledge  of  technical  orders  (17) 

6)  Lack  of  In-Process  Review  experience  (21) 

7)  Lack  of  Statement  of  Work  experience  (27) 

8)  Lack  of  Guidance  Conference  experience  (21) 

9)  Lack  of  tactfulness  in  dealing  with  people  (26) 

10)  Lack  of  cost  evaluation  experience  (30) 

11)  Other  (Please  elaborate  at  the  end  of  questionnaire.) 

(9) 

No  Response  (2) 

12)  What  strong  points  have  you  witnessed  in  other  tech 
order  managers? 

1)  Acquisition  background  (23) 

2)  Configuration  background  (15) 

3)  Knowledge  of  negotiation  techniques  (16) 

4)  Data  Item  Description  experience  (16) 

5)  Knowledge  of  technical  orders  (36) 

6)  In-Process  Review  experience  (23) 

7)  Statement  of  Work  experience  (18) 

8)  Guidance  Conference  experience  (20) 

9)  Tactfulness  in  dealing  with  people  (32) 

10)  Cost  evaluation  experience  (16) 

11)  Other  (Please  elaborate  at  the  end  of  the 
questionnaire.)  (5) 


13)  Please  try  to  list  and  explain,  if  necessary,  all  of 
your  duties  associated  with  technical  order  management  (such 
as  negotiations,  in-process-reviews,  data  item  description 
review,  etc . ) 

In-Process  Reviews  (35) 

Guidance  Conferences  (24) 

Statements  of  Work  (22) 

Valldatlon/Verlf ication  (21) 

Data  Item  Description  (19) 

Negotiation  (19) 

Cost  Evaluation  (10) 

Pre/Post  Publications  Review  (9) 

No  Response  (7) 
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14)  Why  do  you  think  you  were  you  selected  for  this  job? 

Experience/Background  (16) 

Available/Volunteer  (11) 

Maintenance  Background  (8) 

Staff  Experience  (4) 

Permanent  Change  of  Station  (4) 

Luck  ( 4 ) 

Previous  Acquisition  Jobs  (3) 

15)  Is  there  anything  that  you  would  like  to  add  that  might 
help  Improve  the  training  of  tech  order  managers?  If  so, 
please  elaborate. 


16)  What  do  you  think  should  be  included  in  the  criteria  for 
selection  of  tech  order  managers?  (Examples:  field 
experience.  Systems  230,  negotiation  experience,  etc. ) 

Field  Experience  (27) 

Acquisition  Experience  (13) 

Systems  230  (112) 

Rank  (10) 

Negotiations  Experience  (6) 

TOMA  Training  Beforehand  (5) 

Equipment  Knowledge  (4) 
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COMPANY 


LOCATION _ 

INDIVIDUAL _ 

DATE _  TIME _ 

Contractor  Questions 

1)  How  do  you  select  your  TO  managers? 


2)  Are  there  selection  criteria?  If  so,  what  are  they? 


3)  What  type  of  background  for  a  TO  manager  have  you  found 
to  be  most  successful? 
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COMPANY 


4)  What  type  of  training  do  you  provide  for  your  TO 
managers? 


5)  What  experience  or  training  do  you  feel  is  most 
essential  to  the  success  of  a  TO  manager? 


6)  What  are  the  weak  points  of  your  current  selection  and 
training  methods?  Strong  points? 
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